Nov 03 17:16:37.395329 ip-10-0-129-76 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:16:37.802445 ip-10-0-129-76 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:16:37.802445 ip-10-0-129-76 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:16:37.802445 ip-10-0-129-76 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:16:37.802445 ip-10-0-129-76 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:16:37.809956 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809864 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.809956 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809953 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.809956 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809958 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.809956 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809961 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.809956 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809964 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809968 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809974 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809978 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809982 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809985 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809987 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809990 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809993 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809995 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.809998 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810001 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810003 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810006 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810009 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810011 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810015 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810018 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810020 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810023 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.810197 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810025 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810028 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810031 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810033 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810036 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810040 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810042 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810045 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810048 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810050 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810053 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810056 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810059 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810061 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810082 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810086 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810089 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810091 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810094 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810097 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.810673 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810099 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810102 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810105 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810108 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810110 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810113 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810115 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810119 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810122 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810125 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810128 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810130 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810133 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810135 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810140 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810144 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810147 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810150 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810153 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.811163 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810156 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810161 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810163 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810169 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810173 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810177 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810180 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810183 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810186 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810188 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810192 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810195 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810197 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810202 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810206 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810209 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810211 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.811601 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810214 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810218 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810221 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810224 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810226 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810228 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810231 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810234 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810236 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810239 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810242 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810245 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810248 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810251 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810253 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810257 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810262 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810265 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810267 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.812104 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810271 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.812561 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810273 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.812561 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810276 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.812561 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.810278 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.812724 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812710 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812725 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812728 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812731 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812734 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812737 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812740 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812743 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812745 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812748 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812751 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812754 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812757 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.812754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812760 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812763 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812765 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812768 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812771 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812774 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812776 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812779 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812781 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812784 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812786 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812790 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812795 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812798 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812801 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812804 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812807 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812810 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812813 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.813089 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812815 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812818 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812820 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812823 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812825 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812828 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812830 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812833 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812836 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812838 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812841 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812844 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812847 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812849 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812852 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812855 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812858 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812861 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812863 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812866 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.813547 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812868 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812871 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812873 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812876 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812878 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812881 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812883 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812889 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812892 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812894 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812897 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812899 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812902 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812904 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812907 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812909 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812912 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812914 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.814012 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812917 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812923 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812925 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812928 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812931 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812933 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812939 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812942 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812944 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812949 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812952 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812954 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812957 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812959 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812961 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812964 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812966 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.814492 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812969 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812973 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812976 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812980 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812982 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812985 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812988 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812991 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812993 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812996 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.812998 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813001 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813005 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813008 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813010 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813553 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813562 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813565 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813568 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.814992 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813571 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813574 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813577 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813580 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813582 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813585 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813588 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813590 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813593 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813596 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813598 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813601 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813603 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813606 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813608 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813611 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813613 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813615 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813618 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813620 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.815477 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813623 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813625 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813628 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813630 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813634 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813637 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813639 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813642 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813644 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813647 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813650 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813653 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813656 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813658 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813661 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813663 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813666 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813668 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813671 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.815945 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813673 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813676 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813678 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813681 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813683 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813686 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813688 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813691 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813693 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813696 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813698 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813701 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813703 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813706 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813708 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813711 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813713 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813716 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813720 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.816403 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813723 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813725 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813728 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813731 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813734 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813737 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813740 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813743 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813746 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813748 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813750 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813753 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813755 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813758 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813761 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813763 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813766 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813768 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813771 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.816844 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813773 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813780 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813783 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813787 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813791 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813794 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813797 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813800 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813802 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813805 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813807 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813810 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813812 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813816 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813819 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813821 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813824 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.817297 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813827 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813830 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813832 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813834 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.813837 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814710 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814720 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814728 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814733 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814738 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814741 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814746 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814751 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814754 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814757 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814760 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814764 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814767 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814771 2578 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814774 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814777 2578 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814780 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814783 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:16:37.817747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814788 2578 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814792 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814795 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814799 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814802 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814808 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814811 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814815 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814818 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814821 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814825 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814829 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814832 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814835 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814839 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814842 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814845 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814848 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814851 2578 flags.go:64] FLAG: --enable-server="true" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814854 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814858 2578 flags.go:64] FLAG: --event-burst="100" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814862 2578 flags.go:64] FLAG: --event-qps="50" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814865 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814868 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:16:37.818290 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814871 2578 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814875 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814878 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814882 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814885 2578 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814888 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814891 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814894 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814897 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814900 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814903 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814906 2578 flags.go:64] FLAG: --feature-gates="" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814910 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814913 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814916 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814919 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814923 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814926 2578 flags.go:64] FLAG: --help="false" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814929 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814933 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814936 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814939 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814942 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:16:37.818913 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814945 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814948 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814953 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814956 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814959 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814962 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814966 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814969 2578 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814972 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814975 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814978 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814981 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814984 2578 flags.go:64] FLAG: --lock-file="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814987 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814990 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814993 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.814999 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815002 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815005 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815009 2578 flags.go:64] FLAG: --logging-format="text" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815011 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815015 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815018 2578 flags.go:64] FLAG: --manifest-url="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815021 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815026 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:16:37.819471 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815029 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815033 2578 flags.go:64] FLAG: --max-pods="110" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815036 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815039 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815043 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815046 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815049 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815052 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815056 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815081 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815085 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815089 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815092 2578 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815094 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815097 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815100 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815103 2578 flags.go:64] FLAG: --port="10250" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815106 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815109 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c830310aab35c948" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815112 2578 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815115 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815118 2578 flags.go:64] FLAG: --register-node="true" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815121 2578 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815125 2578 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815128 2578 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815131 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:16:37.820084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815133 2578 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815137 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815140 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815143 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815146 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815149 2578 flags.go:64] FLAG: --runonce="false" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815152 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815155 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815158 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815160 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815163 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815167 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815170 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815174 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815176 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815181 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815184 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815186 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815190 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815192 2578 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815195 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815201 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815203 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815206 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815211 2578 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:16:37.820696 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815214 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815217 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815220 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815223 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815226 2578 flags.go:64] FLAG: --v="2" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815230 2578 flags.go:64] FLAG: --version="false" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815235 2578 flags.go:64] FLAG: --vmodule="" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815239 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815242 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815385 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815388 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815392 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815395 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815398 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815400 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815403 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815407 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815409 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815412 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815415 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815417 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.821296 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815420 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815424 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815426 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815429 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815432 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815434 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815437 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815439 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815442 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815445 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815447 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815450 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815452 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815455 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815457 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815460 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815462 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815467 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.821795 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815470 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815472 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815475 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815479 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815482 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815485 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815487 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815490 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815493 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815496 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815498 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815501 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815504 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815507 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815511 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815514 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815516 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815519 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815521 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.822271 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815524 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815527 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815529 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815532 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815534 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815537 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815539 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815542 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815545 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815547 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815550 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815553 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815556 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815558 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815561 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815563 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815567 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815570 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815572 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815575 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.822715 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815577 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815580 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815582 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815586 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815591 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815594 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815598 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815603 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815607 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815610 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815613 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815616 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815619 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815621 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815624 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815626 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815629 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815632 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815634 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.823206 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815637 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815640 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815642 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815645 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815648 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815650 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815653 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815655 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815658 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815662 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815664 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815667 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815671 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815674 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815676 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815679 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.823699 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815681 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.815709 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815754 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815759 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815764 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815767 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815770 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815773 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815776 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815779 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815781 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815784 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815787 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815789 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815792 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815794 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.824091 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815797 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815799 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815802 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815804 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815807 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815810 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815815 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815817 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815820 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815823 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815826 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815829 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815832 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815834 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815837 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815839 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815842 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815845 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.824476 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815847 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815852 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815855 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815858 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815860 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815863 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815865 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815868 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815871 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815873 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815876 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815878 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815881 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815883 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815886 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815888 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815891 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815893 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815896 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815898 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.825006 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815901 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815903 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815905 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815909 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815912 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815915 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815917 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815920 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815922 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815926 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815928 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815931 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815933 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815937 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815940 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815943 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815945 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815948 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815951 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815954 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.825493 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815958 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815962 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815964 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815968 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815971 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815975 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815978 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815980 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815983 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815986 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815988 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815991 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815993 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815996 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.815998 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816002 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816005 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816007 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816010 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.825955 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816012 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816015 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816018 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816020 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816023 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816025 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816029 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816032 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816034 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816037 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816040 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816042 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816045 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.816047 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.822776 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.822793 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.823426 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.823433 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823494 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823500 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.826420 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823503 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823506 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823509 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823512 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823515 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823518 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823520 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823523 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823528 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823546 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823550 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823554 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823557 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823560 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823563 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823565 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823568 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823571 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823574 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.826908 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823576 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823579 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823581 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823584 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823586 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823590 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823593 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823597 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823600 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823602 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823605 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823607 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823610 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823612 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823615 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823617 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823619 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823622 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823624 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.827385 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823627 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823629 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823632 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823634 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823637 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823639 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823642 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823644 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823647 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823649 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823652 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823655 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823659 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823663 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823666 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823668 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823672 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823676 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823679 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.827833 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823682 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823685 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823688 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823691 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823693 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823696 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823698 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823701 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823704 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823706 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823709 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823712 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823714 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823717 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823719 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823722 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823724 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823727 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823729 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823732 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.828285 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823735 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823737 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823740 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823743 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823746 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823749 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823751 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823754 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823756 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823759 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823761 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823764 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823767 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823769 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823772 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823775 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823778 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823781 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823783 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.828781 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823786 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823788 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823791 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823793 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823796 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823798 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823800 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.823828 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823972 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823976 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823979 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823982 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823985 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823988 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823990 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823993 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.829242 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823996 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.823998 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824001 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824004 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824007 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824009 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824012 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824015 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824018 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824020 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824023 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824025 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824028 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824031 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824034 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824037 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824039 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824042 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824044 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.829605 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824047 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824049 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824052 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824054 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824057 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824059 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824075 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824078 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824081 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824083 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824086 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824088 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824092 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824097 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824100 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824103 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824106 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824109 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.830049 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824117 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824122 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824124 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824127 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824130 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824132 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824135 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824138 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824140 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824143 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824145 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824147 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824150 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824152 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824155 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824157 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824160 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824162 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.830528 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824165 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824167 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824170 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824172 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824175 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824177 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824180 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824182 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824185 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824188 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824191 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824193 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824196 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824198 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824201 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824203 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824207 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824211 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824214 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824217 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.830988 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824219 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824222 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824225 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824228 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824230 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824235 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824238 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824240 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824243 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824245 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824248 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824250 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824253 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824255 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824258 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824260 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824265 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.831472 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824268 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.824292 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824323 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824328 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824330 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824333 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824335 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824338 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824340 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824345 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824349 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824352 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824355 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824358 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824360 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824364 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:16:37.831999 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824371 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824376 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824379 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824382 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824384 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824386 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824389 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824392 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824394 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824397 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824399 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824402 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824404 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824407 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824409 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824412 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824414 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:16:37.832569 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824417 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824419 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824422 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824424 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824429 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824432 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824434 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824437 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824439 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824441 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824444 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824447 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824450 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824454 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824457 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824459 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824462 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824464 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824467 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:16:37.833264 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824469 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824472 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824474 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824477 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824479 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824482 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824485 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824488 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824490 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824494 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824497 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824500 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824502 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824505 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824507 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824509 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824512 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824516 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824519 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:16:37.833754 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824521 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824524 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824527 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824530 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824532 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824535 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824543 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824545 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824548 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824550 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824553 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824555 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824558 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824560 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824563 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824565 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824568 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824570 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:16:37.834221 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824573 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824575 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824577 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824583 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824585 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824588 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824590 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824593 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824595 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824598 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824600 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824603 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824606 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824608 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:37.824611 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.824671 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.826835 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.827880 2578 server.go:1014] "Starting client certificate rotation" Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.827974 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:16:37.834871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.828010 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:16:37.835539 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.830208 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:16:37.835539 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.835426 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:16:37.847676 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.847656 2578 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:16:37.851739 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.851721 2578 log.go:25] "Validated CRI v1 image API" Nov 03 17:16:37.852580 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.852558 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:16:37.855951 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.855923 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d7987b8b-663a-463c-a9d0-ed8301acd187:/dev/nvme0n1p3 f0a42282-1e06-4d37-829b-4383f2931079:/dev/nvme0n1p4] Nov 03 17:16:37.856024 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.855949 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 17:16:37.861349 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.861236 2578 manager.go:217] Machine: {Timestamp:2026-11-03 17:16:37.85962377 +0000 UTC m=+0.345236640 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094167 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5737f093a362ad076d637691ddda SystemUUID:ec2c5737-f093-a362-ad07-6d637691ddda BootID:8d992db9-135f-4a60-99bb-a65ea56bf896 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f4:a0:ef:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f4:a0:ef:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:db:e6:d4:32:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:16:37.861433 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.861354 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:16:37.861474 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.861463 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:16:37.865725 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865698 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:16:37.865861 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865727 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-76.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:16:37.865905 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865863 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:16:37.865905 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865871 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:16:37.865905 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865896 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:16:37.865981 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.865910 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:16:37.866784 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.866774 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:16:37.870909 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.870897 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:16:37.871043 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871034 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:16:37.871768 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871758 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:16:37.871798 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871776 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:16:37.871831 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871797 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:16:37.871831 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871810 2578 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:16:37.871831 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.871821 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:16:37.872268 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.872256 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:16:37.874547 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.874530 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:16:37.880606 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.880593 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:16:37.881566 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881553 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:16:37.881603 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881592 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:16:37.881803 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881781 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:16:37.881892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881814 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:16:37.881892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881830 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:16:37.881892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881842 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:16:37.881892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881856 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:16:37.881892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.881867 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:16:37.882358 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882349 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:16:37.882397 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882359 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:16:37.882397 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882376 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:16:37.882397 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882382 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:16:37.882397 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882394 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:16:37.882599 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.882403 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:16:37.883117 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.883108 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:16:37.883155 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.883119 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:16:37.893137 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.893120 2578 server.go:1263] "Started kubelet" Nov 03 17:16:37.897015 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.893148 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:16:37.897195 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.897142 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:16:37.897974 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.897954 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:16:37.898046 ip-10-0-129-76 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:16:37.900240 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.900219 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:16:37.901024 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.900995 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:16:37.903658 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.903633 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 17:16:37.907198 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.907177 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:16:37.907716 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.907701 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:16:37.907835 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.907813 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:37.908653 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.908638 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:16:37.908653 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.908652 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:16:37.908790 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.908686 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:16:37.908790 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.908748 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:16:37.908790 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.908756 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:16:37.909507 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.909334 2578 factory.go:55] Registering systemd factory Nov 03 17:16:37.909507 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.909370 2578 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:16:37.910089 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.910055 2578 factory.go:153] Registering CRI-O factory Nov 03 17:16:37.910157 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.910094 2578 factory.go:223] Registration of the crio container factory successfully Nov 03 17:16:37.910157 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.910140 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:16:37.910255 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.910159 2578 factory.go:103] Registering Raw factory Nov 03 17:16:37.910255 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.910174 2578 manager.go:1199] Started watching for new ooms in manager Nov 03 17:16:37.911022 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.911011 2578 manager.go:319] Starting recovery of all containers Nov 03 17:16:37.917603 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.917553 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:16:37.921035 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.921020 2578 manager.go:324] Recovery completed Nov 03 17:16:37.926057 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.926035 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:37.928054 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928035 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:37.928054 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928080 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:37.928196 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928091 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:37.928615 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928601 2578 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:16:37.928657 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928616 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:16:37.928657 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928640 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:16:37.928716 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.928654 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:37.929391 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.929378 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:37.929439 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.929407 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:37.929439 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.929419 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:37.930889 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.930879 2578 policy_none.go:50] "Start" Nov 03 17:16:37.930922 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.930893 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:16:37.930922 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.930903 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:16:37.932973 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.932964 2578 policy_none.go:44] "Start" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.977661 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.981501 2578 manager.go:341] "Starting Device Plugin manager" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.982114 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982128 2578 server.go:87] "Starting device plugin registration server" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982357 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982367 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982452 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982539 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.982546 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.983103 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.983146 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.985174 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.985195 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:37.985210 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:16:37.989351 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:37.985239 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 17:16:38.024123 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.024096 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-76.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 17:16:38.069210 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.069104 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-76.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 17:16:38.082904 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.082886 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.084540 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.084525 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.084609 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.084557 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.084609 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.084580 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.084609 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.084605 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.085493 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.085459 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal"] Nov 03 17:16:38.085580 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.085539 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.086292 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.086273 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.086363 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.086303 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.086363 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.086317 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.086609 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.086583 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 17:16:38.087673 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.086599 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.us-west-2.compute.internal.18e4987931b960ca default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:16:37.892997322 +0000 UTC m=+0.378610193,LastTimestamp:2026-11-03 17:16:37.892997322 +0000 UTC m=+0.378610193,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.us-west-2.compute.internal,}" Nov 03 17:16:38.088603 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.088588 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.088759 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.088743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.088806 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.088775 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.089342 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089317 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.089342 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089340 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.089452 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089363 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.089452 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089339 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.089452 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089442 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.089594 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.089458 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.092220 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.092197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.092220 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.092221 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.092882 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.092860 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.092963 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.092892 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.092963 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.092906 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.109204 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.109182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.109325 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.109216 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.109325 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.109242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.121738 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.121719 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.123905 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.123889 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.142773 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.142749 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:16:38.142882 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.142821 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8nvsq" Nov 03 17:16:38.202264 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.202152 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-76.us-west-2.compute.internal.18e4987933d04831 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-76.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:16:37.928052785 +0000 UTC m=+0.413665655,LastTimestamp:2026-11-03 17:16:37.928052785 +0000 UTC m=+0.413665655,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.us-west-2.compute.internal,}" Nov 03 17:16:38.202264 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.202246 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-76.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 17:16:38.209875 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.209976 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.209976 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.209976 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209971 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.210116 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.210116 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.209970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.252105 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.252013 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 17:16:38.252105 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.252093 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 17:16:38.269259 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.269235 2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.379728 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.379670 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.425233 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.425206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.428785 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.428761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.469540 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.469511 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:38.470475 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.470456 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:38.470566 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.470488 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:38.470566 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.470498 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:38.470566 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.470528 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.530021 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.529996 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:38.530125 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.530026 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-76.us-west-2.compute.internal\": node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:38.662772 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.662685 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:38.763376 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.763343 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:38.827965 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:38.827931 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 17:16:38.828510 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.828183 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae4f21c5985754fdab24e2f2e3e09060-2a5fa75794b3f366.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-76.us-west-2.compute.internal.18e4987933d0ca02\": read tcp 10.0.129.76:33792->3.133.66.197:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-76.us-west-2.compute.internal.18e4987933d0ca02 default 5785 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-76.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-76.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-76.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:16:37 +0000 UTC,LastTimestamp:2026-11-03 17:16:38.084561399 +0000 UTC m=+0.570174270,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-76.us-west-2.compute.internal,}" Nov 03 17:16:38.864254 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.864228 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:38.965049 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:38.964990 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:39.065205 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:39.065175 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:39.148101 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.148039 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:11:37 +0000 UTC" deadline="2028-07-06 18:57:47.091786757 +0000 UTC" Nov 03 17:16:39.148101 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.148092 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14665h41m7.943698974s" Nov 03 17:16:39.165292 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:39.165264 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:39.211006 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.210979 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:16:39.211863 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.211843 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:16:39.211951 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.211874 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:16:39.211951 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.211884 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:16:39.218856 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.218813 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:16:39.265739 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:39.265703 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:39.366672 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:39.366645 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:16:39.388494 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.388468 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:16:39.388652 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.388500 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:16:39.408932 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.408783 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:39.473335 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.473289 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:16:39.474282 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.474269 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:16:39.511122 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:39.511090 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fb60344eae41808f24a7b8d8670170.slice/crio-9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794 WatchSource:0}: Error finding container 9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794: Status 404 returned error can't find the container with id 9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794 Nov 03 17:16:39.511457 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:39.511435 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9034c461ae30eb2c88e2ba1d28d459.slice/crio-1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb WatchSource:0}: Error finding container 1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb: Status 404 returned error can't find the container with id 1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb Nov 03 17:16:39.516296 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.516282 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:16:39.563515 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.563491 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:16:39.820627 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.820556 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:16:39.873406 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.873383 2578 apiserver.go:52] "Watching apiserver" Nov 03 17:16:39.908287 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.908269 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 17:16:39.927344 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.927320 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:16:39.927505 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.927490 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xw9w5","openshift-network-diagnostics/network-check-target-j65bb","openshift-network-operator/iptables-alerter-wptbt","kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal"] Nov 03 17:16:39.930857 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.930842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:39.932675 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.932647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:39.932789 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:39.932733 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:39.934486 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.934472 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:39.936494 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.936394 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:39.980553 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.980534 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h78gf\"" Nov 03 17:16:39.988094 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.988048 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czdwp\"" Nov 03 17:16:39.988734 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.988699 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"31fb60344eae41808f24a7b8d8670170","Type":"ContainerStarted","Data":"9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794"} Nov 03 17:16:39.989010 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.988996 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:16:39.989302 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.989287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ptg9\"" Nov 03 17:16:39.989686 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:39.989671 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb"} Nov 03 17:16:40.009139 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.009120 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:16:40.019348 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019423 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019423 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019499 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019499 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019597 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019597 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019597 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019587 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019701 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:40.019701 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019642 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.019701 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019701 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp9n\" (UniqueName: \"kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2bl\" (UniqueName: \"kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.019876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.019862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvsv\" (UniqueName: \"kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.046135 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046111 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:16:40.046270 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046187 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:16:40.046270 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046259 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:16:40.046351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046274 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:16:40.046382 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046353 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:16:40.046526 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046508 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:16:40.046526 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:16:40.046654 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046547 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:16:40.046654 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046613 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:16:40.046791 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.046777 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:16:40.058206 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.058185 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:16:40.058292 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.058246 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q4vfc" Nov 03 17:16:40.120880 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvsv\" (UniqueName: \"kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121012 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.120987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121089 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121154 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp9n\" (UniqueName: \"kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121351 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2bl\" (UniqueName: \"kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121402 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121440 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.121870 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.122085 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.121915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.178271 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.178239 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:16:40.180636 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.180609 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:40.180636 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.180638 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:40.180793 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.180652 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:40.180793 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.180719 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:40.680700156 +0000 UTC m=+3.166313034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:40.181638 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.181618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2bl\" (UniqueName: \"kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.182126 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.182112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp9n\" (UniqueName: \"kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.184948 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.184934 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvsv\" (UniqueName: \"kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.252442 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.252407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:16:40.257169 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.257153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:16:40.260605 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.260589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:16:40.261633 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:40.261615 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8b9a87_1619_4d2c_8989_3bba620ce5b5.slice/crio-caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172 WatchSource:0}: Error finding container caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172: Status 404 returned error can't find the container with id caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172 Nov 03 17:16:40.266706 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:40.266682 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6141b3_1493_4bd1_b637_5d4290f46930.slice/crio-d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9 WatchSource:0}: Error finding container d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9: Status 404 returned error can't find the container with id d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9 Nov 03 17:16:40.271160 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:16:40.271144 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39eb672_c5af_4f63_b7ef_b34fa3674c6d.slice/crio-50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076 WatchSource:0}: Error finding container 50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076: Status 404 returned error can't find the container with id 50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076 Nov 03 17:16:40.724246 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.724212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:40.724409 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.724339 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:40.724409 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.724356 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:40.724409 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.724365 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:40.724529 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:40.724418 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:41.724401144 +0000 UTC m=+4.210014001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:40.993081 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.992982 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" event={"ID":"f39eb672-c5af-4f63-b7ef-b34fa3674c6d","Type":"ContainerStarted","Data":"50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076"} Nov 03 17:16:40.996694 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.996622 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerStarted","Data":"d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9"} Nov 03 17:16:40.998159 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:40.998124 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wptbt" event={"ID":"bb8b9a87-1619-4d2c-8989-3bba620ce5b5","Type":"ContainerStarted","Data":"caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172"} Nov 03 17:16:41.734094 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:41.732636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:41.734094 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:41.732809 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:41.734094 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:41.732837 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:41.734094 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:41.732852 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:41.734094 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:41.732910 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:43.73289145 +0000 UTC m=+6.218504324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:41.985784 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:41.985726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:41.985948 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:41.985852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:42.001498 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:42.001460 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91"} Nov 03 17:16:42.003360 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:42.003329 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"31fb60344eae41808f24a7b8d8670170","Type":"ContainerStarted","Data":"7f8bce6e88cdc55735248921950b2bc6b830aa471edb165cdc8d1c4ef4942ea8"} Nov 03 17:16:42.240359 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:42.240041 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" podStartSLOduration=3.240022727 podStartE2EDuration="3.240022727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:42.239687356 +0000 UTC m=+4.725300235" watchObservedRunningTime="2026-11-03 17:16:42.240022727 +0000 UTC m=+4.725635606" Nov 03 17:16:43.007768 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:43.007729 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="a469eb15a493fb0399a22d31c9f9d3de6accf2a8ec527f0c0acc1f4724814860" exitCode=0 Nov 03 17:16:43.008249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:43.007811 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"a469eb15a493fb0399a22d31c9f9d3de6accf2a8ec527f0c0acc1f4724814860"} Nov 03 17:16:43.746369 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:43.746262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:43.746507 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:43.746416 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:43.746507 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:43.746434 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:43.746507 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:43.746446 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:43.746631 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:43.746508 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:47.746489961 +0000 UTC m=+10.232102819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:43.986413 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:43.986379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:43.986574 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:43.986486 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:44.010739 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:44.010670 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" event={"ID":"f39eb672-c5af-4f63-b7ef-b34fa3674c6d","Type":"ContainerStarted","Data":"85fd12eb9b48eca2b09e5c37f75f061cef90e483b2c99bc3e7ac1c94bd954a90"} Nov 03 17:16:45.247458 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:45.247421 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:16:45.986284 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:45.986189 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:16:45.247442919Z","UUID":"ad7ec0a4-6fc9-4d3b-9375-c1f4e7327375","Handler":null,"Name":"","Endpoint":""} Nov 03 17:16:45.986284 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:45.986265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:45.986486 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:45.986364 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:45.987540 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:45.987518 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:16:45.987665 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:45.987547 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:16:46.014958 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:46.014930 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" event={"ID":"f39eb672-c5af-4f63-b7ef-b34fa3674c6d","Type":"ContainerStarted","Data":"aaee4896509760cd8dbd2273719e02a2c2ef620efcb3c8710d4a1257e7ae2d08"} Nov 03 17:16:46.016666 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:46.016644 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="cc4706d9ca8c315dd047036685faec0eec1fc89a6d41f4e58000d88f67f8a739" exitCode=0 Nov 03 17:16:46.016747 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:46.016713 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"cc4706d9ca8c315dd047036685faec0eec1fc89a6d41f4e58000d88f67f8a739"} Nov 03 17:16:46.018080 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:46.018044 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wptbt" event={"ID":"bb8b9a87-1619-4d2c-8989-3bba620ce5b5","Type":"ContainerStarted","Data":"cebc0cd7ce0ac894f15c31f0c65febfcbd5eac5fdd40d05d646c02ad282af772"} Nov 03 17:16:47.021759 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:47.021665 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" event={"ID":"f39eb672-c5af-4f63-b7ef-b34fa3674c6d","Type":"ContainerStarted","Data":"56cf606d5437cb5dd573ed8b440d8488587a6701ff2196368a1df29f0513ef61"} Nov 03 17:16:47.142020 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:47.141966 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wptbt" podStartSLOduration=5.356397856 podStartE2EDuration="9.141945214s" totalImagesPullingTime="3.785547358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:46.284284277 +0000 UTC m=+8.769897153" watchObservedRunningTime="2026-11-03 17:16:47.141945214 +0000 UTC m=+9.627558094" Nov 03 17:16:47.142199 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:47.142112 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" podStartSLOduration=2.968315725 podStartE2EDuration="9.142105897s" totalImagesPullingTime="6.173790172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:47.142054019 +0000 UTC m=+9.627666897" watchObservedRunningTime="2026-11-03 17:16:47.142105897 +0000 UTC m=+9.627718775" Nov 03 17:16:47.768025 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:47.767992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:47.768227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:47.768118 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:47.768227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:47.768142 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:47.768227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:47.768151 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:47.768227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:47.768199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:55.768186268 +0000 UTC m=+18.253799139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:47.985720 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:47.985682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:47.985865 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:47.985787 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:48.024866 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:48.024783 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="70c6a7cf4fc203052f18dbe1150a825a6856086adda4f4cdbaf6e5ef00816cee" exitCode=0 Nov 03 17:16:48.025313 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:48.024866 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"70c6a7cf4fc203052f18dbe1150a825a6856086adda4f4cdbaf6e5ef00816cee"} Nov 03 17:16:49.986197 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:49.986165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:49.986638 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:49.986283 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:50.030554 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:50.030516 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="87d26756fdf80db61efe42d933e6d36484aae865e29bd6c9690ac5318cb1d4e5" exitCode=0 Nov 03 17:16:50.030711 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:50.030561 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"87d26756fdf80db61efe42d933e6d36484aae865e29bd6c9690ac5318cb1d4e5"} Nov 03 17:16:51.986131 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:51.986109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:51.986434 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:51.986234 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:52.037773 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:52.037741 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerStarted","Data":"35a746bd7e1638e0512cb0211e7468643abd2cf7f5fb6a2bde990c289187079e"} Nov 03 17:16:53.041396 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.041366 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_bf9034c461ae30eb2c88e2ba1d28d459/setup/0.log" Nov 03 17:16:53.041849 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.041405 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf9034c461ae30eb2c88e2ba1d28d459" containerID="caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91" exitCode=1 Nov 03 17:16:53.041849 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.041473 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerDied","Data":"caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91"} Nov 03 17:16:53.044366 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.044339 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="35a746bd7e1638e0512cb0211e7468643abd2cf7f5fb6a2bde990c289187079e" exitCode=0 Nov 03 17:16:53.044469 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.044378 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"35a746bd7e1638e0512cb0211e7468643abd2cf7f5fb6a2bde990c289187079e"} Nov 03 17:16:53.985758 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:53.985596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:53.985906 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:53.985831 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:54.047858 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:54.047831 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_bf9034c461ae30eb2c88e2ba1d28d459/setup/0.log" Nov 03 17:16:54.048253 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:54.047916 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"3a7994358f7fee92a17f84628cf3a90399c28ac31a7347c1c69952894caddbd3"} Nov 03 17:16:54.050627 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:54.050605 2578 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="f67302a155cda0e970af4aeb435de933bd3b5b0ff0f22c9077d623703d01413e" exitCode=0 Nov 03 17:16:54.050735 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:54.050635 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerDied","Data":"f67302a155cda0e970af4aeb435de933bd3b5b0ff0f22c9077d623703d01413e"} Nov 03 17:16:55.816242 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:55.816204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:55.816721 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:55.816333 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:16:55.816721 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:55.816347 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:16:55.816721 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:55.816360 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:55.816721 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:55.816412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:11.816399045 +0000 UTC m=+34.302011902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:16:55.985576 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:55.985545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:55.985730 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:55.985676 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:57.985843 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:57.985760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:57.986358 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:57.985846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:16:58.060609 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:58.060577 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" event={"ID":"2d6141b3-1493-4bd1-b637-5d4290f46930","Type":"ContainerStarted","Data":"fb9dafa8f32b43c6c4e4a74f912dcf02d3d5056db104bee1a6f81e92031ed995"} Nov 03 17:16:58.287160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:58.287041 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xw9w5" podStartSLOduration=8.387808501 podStartE2EDuration="20.287027769s" totalImagesPullingTime="11.899219268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:38 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:58.286810022 +0000 UTC m=+20.772422902" watchObservedRunningTime="2026-11-03 17:16:58.287027769 +0000 UTC m=+20.772640649" Nov 03 17:16:59.985703 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:16:59.985662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:16:59.986112 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:16:59.985803 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:00.042596 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.042566 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q4vfc" Nov 03 17:17:00.085636 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.085608 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-dw5h9"] Nov 03 17:17:00.087291 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.087276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.094240 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.094220 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-lz759"] Nov 03 17:17:00.095720 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.095704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.115391 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.115372 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-l54gp"] Nov 03 17:17:00.117473 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.117460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.117532 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.117515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:00.139271 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139337 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139337 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139337 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139337 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139465 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.139465 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139536 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqgfm\" (UniqueName: \"kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139536 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.139602 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139602 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139602 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139698 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.139698 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139637 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139698 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139698 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.139828 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.139749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.147134 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.147111 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:17:00.147238 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.147186 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:17:00.147238 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.147200 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:17:00.160365 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.160347 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:17:00.160428 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.160383 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h6tl4\"" Nov 03 17:17:00.166280 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.166266 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:17:00.168245 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.168228 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whvtr\"" Nov 03 17:17:00.199024 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.199005 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bg64f"] Nov 03 17:17:00.200808 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.200794 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-tvt2q"] Nov 03 17:17:00.200914 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.200901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.202236 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.202222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.230462 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.230429 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z526b"] Nov 03 17:17:00.232010 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.231997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.232082 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.232050 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:00.240281 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240281 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240273 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240404 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240404 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240404 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240533 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240533 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240533 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240533 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5n4\" (UniqueName: \"kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.240717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240733 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240799 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.240907 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqhl\" (UniqueName: \"kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.240931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgfm\" (UniqueName: \"kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241249 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241479 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.241871 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241917 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.241961 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7b5m\" (UniqueName: \"kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242153 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242285 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.242527 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.242424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.245614 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.245596 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.245659 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.245637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.255259 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.255240 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xlbvs"] Nov 03 17:17:00.256965 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.256952 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.260926 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.260911 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-xt562"] Nov 03 17:17:00.262492 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.262476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.266766 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.266749 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:17:00.267006 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.266990 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:17:00.267058 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.266993 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:17:00.267058 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.267026 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:17:00.267154 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.267104 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:17:00.267245 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.267222 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:17:00.303674 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.303651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.303776 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.303710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgfm\" (UniqueName: \"kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.311928 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.311914 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:17:00.318417 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.318405 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:17:00.318474 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.318464 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:17:00.342543 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342543 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342632 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342649 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342632 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342672 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342682 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342684 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.342751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5n4\" (UniqueName: \"kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342876 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.342995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.343312 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqhl\" (UniqueName: \"kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343329 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343549 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344133 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343637 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.344687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7b5m\" (UniqueName: \"kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.343998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.344004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.344039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6zw\" (UniqueName: \"kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.343980 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.344006 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.344131 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:00.844114044 +0000 UTC m=+23.329726921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:00.345189 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.344281 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.350118 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350097 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6dxq\"" Nov 03 17:17:00.350118 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350103 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:17:00.350226 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350116 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lkll\"" Nov 03 17:17:00.350226 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350153 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmcvz\"" Nov 03 17:17:00.350226 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350174 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:17:00.350226 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.350193 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwgrr\"" Nov 03 17:17:00.356414 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.356397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.397088 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.397045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:00.404637 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.404617 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:00.405566 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.405544 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c806b4_ceab_4062_aa31_e0776a71ed05.slice/crio-33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d WatchSource:0}: Error finding container 33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d: Status 404 returned error can't find the container with id 33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d Nov 03 17:17:00.410914 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.410889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqhl\" (UniqueName: \"kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.411036 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.411017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5n4\" (UniqueName: \"kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.411164 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.411144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7b5m\" (UniqueName: \"kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.411902 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.411884 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca30f271_2ab1_468f_8413_64c76f5eb172.slice/crio-e4c82584f17d9580518e6f58af90d31fdb0dfc7ee154d21f2d3c2d8e7ad543fe WatchSource:0}: Error finding container e4c82584f17d9580518e6f58af90d31fdb0dfc7ee154d21f2d3c2d8e7ad543fe: Status 404 returned error can't find the container with id e4c82584f17d9580518e6f58af90d31fdb0dfc7ee154d21f2d3c2d8e7ad543fe Nov 03 17:17:00.444890 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.444852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.445041 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.444906 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6zw\" (UniqueName: \"kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.445041 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.444932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.445041 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.444949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.445216 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.445461 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.445534 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.445534 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445497 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.445534 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445545 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.445567 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.445605 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:00.445655 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.445649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:00.945635718 +0000 UTC m=+23.431248579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:00.447782 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.447766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.525980 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.525914 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tvt2q" Nov 03 17:17:00.527768 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.527746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6zw\" (UniqueName: \"kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.528396 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.528376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:00.535856 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.535814 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53025bb_d955_4dd9_a75e_74ad4bdaf61b.slice/crio-d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b WatchSource:0}: Error finding container d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b: Status 404 returned error can't find the container with id d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b Nov 03 17:17:00.536696 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.536674 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6742db_d55d_4aac_b612_dafe9c875cc5.slice/crio-e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0 WatchSource:0}: Error finding container e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0: Status 404 returned error can't find the container with id e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0 Nov 03 17:17:00.565811 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.565788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:00.571640 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.571619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:00.575906 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.575883 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54863910_20c1_44fe_b7f7_f2a39321a9eb.slice/crio-7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298 WatchSource:0}: Error finding container 7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298: Status 404 returned error can't find the container with id 7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298 Nov 03 17:17:00.581579 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:00.581560 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35b8f38_9b01_4c09_85d8_560e93cd71da.slice/crio-72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99 WatchSource:0}: Error finding container 72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99: Status 404 returned error can't find the container with id 72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99 Nov 03 17:17:00.849397 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.849307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:00.849543 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.849450 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:00.849543 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.849512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:01.849497008 +0000 UTC m=+24.335109866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:00.950473 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:00.950441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:00.950668 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.950610 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:00.950728 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:00.950684 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:01.950665606 +0000 UTC m=+24.436278465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:01.044575 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.044510 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:59 +0000 UTC" deadline="2028-07-01 11:38:43.715418642 +0000 UTC" Nov 03 17:17:01.044575 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.044566 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14538h21m42.670855483s" Nov 03 17:17:01.068057 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.067882 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt562" event={"ID":"e35b8f38-9b01-4c09-85d8-560e93cd71da","Type":"ContainerStarted","Data":"a1e6ed2cf6d46cc4789e3d5a046f4e2a1cd0023d370a444314d61ec7bbe9c4a5"} Nov 03 17:17:01.068196 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.068124 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xt562" event={"ID":"e35b8f38-9b01-4c09-85d8-560e93cd71da","Type":"ContainerStarted","Data":"72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99"} Nov 03 17:17:01.069120 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.069095 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0"} Nov 03 17:17:01.070082 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.070051 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lz759" event={"ID":"ca30f271-2ab1-468f-8413-64c76f5eb172","Type":"ContainerStarted","Data":"e4c82584f17d9580518e6f58af90d31fdb0dfc7ee154d21f2d3c2d8e7ad543fe"} Nov 03 17:17:01.071049 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.071029 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw5h9" event={"ID":"37c806b4-ceab-4062-aa31-e0776a71ed05","Type":"ContainerStarted","Data":"33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d"} Nov 03 17:17:01.072665 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.072649 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_bf9034c461ae30eb2c88e2ba1d28d459/setup/0.log" Nov 03 17:17:01.072742 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.072678 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf9034c461ae30eb2c88e2ba1d28d459" containerID="3a7994358f7fee92a17f84628cf3a90399c28ac31a7347c1c69952894caddbd3" exitCode=0 Nov 03 17:17:01.072742 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.072727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerDied","Data":"3a7994358f7fee92a17f84628cf3a90399c28ac31a7347c1c69952894caddbd3"} Nov 03 17:17:01.072842 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.072772 2578 scope.go:122] "RemoveContainer" containerID="caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91" Nov 03 17:17:01.073059 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.073040 2578 scope.go:122] "RemoveContainer" containerID="caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91" Nov 03 17:17:01.073915 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.073895 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xlbvs" event={"ID":"54863910-20c1-44fe-b7f7-f2a39321a9eb","Type":"ContainerStarted","Data":"7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298"} Nov 03 17:17:01.075404 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.075388 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tvt2q" event={"ID":"b53025bb-d955-4dd9-a75e-74ad4bdaf61b","Type":"ContainerStarted","Data":"2e89ca656f56305f21f654f5e6f8afdfb4d4e273f909fc2490cf91aafab3b38a"} Nov 03 17:17:01.075475 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.075420 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tvt2q" event={"ID":"b53025bb-d955-4dd9-a75e-74ad4bdaf61b","Type":"ContainerStarted","Data":"d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b"} Nov 03 17:17:01.083501 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.083474 2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_openshift-machine-config-operator_bf9034c461ae30eb2c88e2ba1d28d459_0 in pod sandbox 1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb from index: no such id: 'caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91'" containerID="caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91" Nov 03 17:17:01.083578 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.083524 2578 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_openshift-machine-config-operator_bf9034c461ae30eb2c88e2ba1d28d459_0 in pod sandbox 1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb from index: no such id: 'caa93ecbea4f7bbc92ef3a8524008f2b3c8acf4dd9e0cb16cd33540492ff0b91'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal_openshift-machine-config-operator(bf9034c461ae30eb2c88e2ba1d28d459)\"" logger="UnhandledError" Nov 03 17:17:01.187136 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.187079 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-xt562" podStartSLOduration=1.187048451 podStartE2EDuration="1.187048451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:01.186879216 +0000 UTC m=+23.672492094" watchObservedRunningTime="2026-11-03 17:17:01.187048451 +0000 UTC m=+23.672661329" Nov 03 17:17:01.307857 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.307803 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tvt2q" podStartSLOduration=1.307789553 podStartE2EDuration="1.307789553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:01.307278534 +0000 UTC m=+23.792891412" watchObservedRunningTime="2026-11-03 17:17:01.307789553 +0000 UTC m=+23.793402431" Nov 03 17:17:01.858468 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.857885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:01.858468 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.858040 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:01.858468 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.858120 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:03.858103075 +0000 UTC m=+26.343715944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:01.958840 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.958735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:01.959018 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.958961 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:01.959104 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.959092 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:03.959027988 +0000 UTC m=+26.444640862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:01.986227 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.986191 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:01.986401 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.986321 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:01.986473 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.986426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:01.986545 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.986520 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:01.987845 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:01.987823 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:01.998327 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:01.998245 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:02.045292 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:02.045235 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:59 +0000 UTC" deadline="2028-05-08 02:32:20.820000561 +0000 UTC" Nov 03 17:17:02.045292 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:02.045288 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13233h15m18.774716567s" Nov 03 17:17:02.082102 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:02.082037 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"fd4690c918573c583e31619b50a8a4b6771d4b046c180e20eac576f66a132fe1"} Nov 03 17:17:02.202148 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:02.202035 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" podStartSLOduration=23.202019522 podStartE2EDuration="23.202019522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:02.201139964 +0000 UTC m=+24.686752845" watchObservedRunningTime="2026-11-03 17:17:02.202019522 +0000 UTC m=+24.687632404" Nov 03 17:17:03.874903 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:03.874871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:03.875334 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.874997 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:03.875334 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.875057 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:07.875038306 +0000 UTC m=+30.360651165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:03.975559 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:03.975518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:03.975725 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.975664 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:03.975725 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.975721 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:07.975703781 +0000 UTC m=+30.461316655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:03.987274 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:03.987200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:03.987274 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:03.987222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:03.987456 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.987332 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:03.987487 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.987456 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:03.987724 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:03.987574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:03.987724 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:03.987653 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:04.095339 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:04.095090 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xlbvs" event={"ID":"54863910-20c1-44fe-b7f7-f2a39321a9eb","Type":"ContainerStarted","Data":"600d5b80c1cbac5464ca9fb134c0e10209a75cb33d436822638390104ab05089"} Nov 03 17:17:04.097321 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:04.097288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dw5h9" event={"ID":"37c806b4-ceab-4062-aa31-e0776a71ed05","Type":"ContainerStarted","Data":"964a4141a11cf53cef213dcb42a04353d13a605a72d91bf20da5fda5de120e78"} Nov 03 17:17:04.213178 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:04.213100 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xlbvs" podStartSLOduration=1.8297962719999998 podStartE2EDuration="4.213086304s" totalImagesPullingTime="2.383290032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:04.213002591 +0000 UTC m=+26.698615481" watchObservedRunningTime="2026-11-03 17:17:04.213086304 +0000 UTC m=+26.698699179" Nov 03 17:17:04.331013 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:04.330958 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dw5h9" podStartSLOduration=0.950424727 podStartE2EDuration="4.330943095s" totalImagesPullingTime="3.380518368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:04.330124996 +0000 UTC m=+26.815737876" watchObservedRunningTime="2026-11-03 17:17:04.330943095 +0000 UTC m=+26.816555980" Nov 03 17:17:05.566362 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:05.566274 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:05.566875 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:05.566853 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:05.987168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:05.987300 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:05.987819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:05.987941 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:05.988011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:05.988227 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:05.988180 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:07.106267 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.106183 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lz759" event={"ID":"ca30f271-2ab1-468f-8413-64c76f5eb172","Type":"ContainerStarted","Data":"76c641a704c3fc7a8275ced3fe21494ceb5b6e9d544f4feca91bda2efc56b1c3"} Nov 03 17:17:07.106267 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.106212 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:17:07.903970 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.903927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:07.904163 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:07.904121 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:07.904210 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:07.904198 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:15.90417947 +0000 UTC m=+38.389792330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:07.986876 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.986841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:07.987035 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.986937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:07.987035 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:07.986986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:07.987035 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:07.987019 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:07.987179 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:07.987060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:07.987179 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:07.987157 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:08.004374 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:08.004348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:08.004501 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:08.004444 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:08.004501 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:08.004493 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:16.004480681 +0000 UTC m=+38.490093537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:09.986572 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:09.986532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:09.987028 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:09.986600 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:09.987028 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:09.986611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:09.987028 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:09.986696 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:09.987224 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:09.987048 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:09.987224 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:09.987130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:11.119242 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119048 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"a77459f6c65a54e21e343c5387fb6f00dd4693e091e09a47db9e556e4f91f7fc"} Nov 03 17:17:11.119632 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119323 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"c95392a592490dca901f1ec8d7017ab51053ed3c4789365932d21981b5f10524"} Nov 03 17:17:11.119632 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119357 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"afd71bb80c1e0fb6f1ecfc4ccf7bf1bd278c96b1116f66488bf25db6e5cab346"} Nov 03 17:17:11.119632 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119407 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"fee26cd8ad98573ce7d2f53245eb87e467625ce8989d8bbed8b18314fb5e6287"} Nov 03 17:17:11.119632 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119435 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"c88a62efd235f814859e966e49d8edb39a3a51718167fae93371565c84cb67b8"} Nov 03 17:17:11.119632 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.119462 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"4b8c8b562ec3177f4de2a8567d2fa72a2e7e539986f0af7d5ffc22017c9bffd4"} Nov 03 17:17:11.834432 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.834395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:11.834617 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.834590 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:17:11.834617 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.834613 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:17:11.834733 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.834625 2578 projected.go:196] Error preparing data for projected volume kube-api-access-4mx59 for pod openshift-network-diagnostics/network-check-target-j65bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:17:11.834733 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.834698 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59 podName:83f65e07-2574-44d8-9fe0-de9b7f689bd4 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:43.834667227 +0000 UTC m=+66.320280098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mx59" (UniqueName: "kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59") pod "network-check-target-j65bb" (UID: "83f65e07-2574-44d8-9fe0-de9b7f689bd4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:17:11.986303 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.986267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:11.986303 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.986289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:11.986532 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.986433 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:11.986532 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:11.986482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:11.986640 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.986559 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:11.986640 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:11.986631 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:12.751179 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.751122 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lz759" podStartSLOduration=6.473479607 podStartE2EDuration="12.751109202s" totalImagesPullingTime="6.277629595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:07.231168508 +0000 UTC m=+29.716781388" watchObservedRunningTime="2026-11-03 17:17:12.751109202 +0000 UTC m=+35.236722080" Nov 03 17:17:12.751672 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.751298 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8mxk9"] Nov 03 17:17:12.755418 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.755396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.825090 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.825043 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 17:17:12.839400 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839507 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839507 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839594 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839653 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczxl\" (UniqueName: \"kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839687 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839751 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.839816 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.839798 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.861149 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861122 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vh8j\"" Nov 03 17:17:12.861275 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861157 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 17:17:12.861275 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861166 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:17:12.861275 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861202 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:17:12.861559 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861543 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 17:17:12.861621 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.861602 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 17:17:12.940620 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940620 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940818 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940818 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczxl\" (UniqueName: \"kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940818 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940949 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940949 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940949 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940949 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.940949 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.941181 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.940965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.941181 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.941042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.941181 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.941094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.941419 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.941399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.941465 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.941429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.944620 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.944600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:12.944715 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:12.944607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:13.003389 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.003314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczxl\" (UniqueName: \"kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:13.072960 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.072924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:13.084572 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:13.084548 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b90722_afde_44ca_88ac_a73e3b3c61aa.slice/crio-9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873 WatchSource:0}: Error finding container 9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873: Status 404 returned error can't find the container with id 9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873 Nov 03 17:17:13.123674 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.123647 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mxk9" event={"ID":"b7b90722-afde-44ca-88ac-a73e3b3c61aa","Type":"ContainerStarted","Data":"9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873"} Nov 03 17:17:13.986154 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.986101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:13.986154 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.986135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:13.986904 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:13.986112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:13.986904 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:13.986249 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:13.986904 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:13.986293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:13.986904 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:13.986356 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:14.130807 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:14.130763 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"6cdeae4bc3b6c9af63e8e03531008b886c03a1f60221bf1dc56a32769c76f956"} Nov 03 17:17:15.134425 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.134392 2578 generic.go:356] "Generic (PLEG): container finished" podID="b7b90722-afde-44ca-88ac-a73e3b3c61aa" containerID="5ff053078609eab0f476adc6cfa64bfa5cc68b5af5bf4ded2b36e9c67bb398b4" exitCode=0 Nov 03 17:17:15.134862 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.134456 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mxk9" event={"ID":"b7b90722-afde-44ca-88ac-a73e3b3c61aa","Type":"ContainerDied","Data":"5ff053078609eab0f476adc6cfa64bfa5cc68b5af5bf4ded2b36e9c67bb398b4"} Nov 03 17:17:15.962917 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.962737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:15.962917 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:15.962901 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:15.963163 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:15.962970 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs podName:266c9430-4b51-4065-a2fa-00f638011a0f nodeName:}" failed. No retries permitted until 2026-11-03 17:17:31.962954405 +0000 UTC m=+54.448567288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs") pod "network-metrics-daemon-l54gp" (UID: "266c9430-4b51-4065-a2fa-00f638011a0f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:17:15.988625 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.988595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:15.988739 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.988710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:15.988739 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:15.988721 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:15.988739 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:15.988737 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:15.988858 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:15.988809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:15.988900 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:15.988866 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:16.063750 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.063720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:16.063899 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:16.063850 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:16.063942 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:16.063901 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret podName:9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb nodeName:}" failed. No retries permitted until 2026-11-03 17:17:32.06388908 +0000 UTC m=+54.549501937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret") pod "global-pull-secret-syncer-z526b" (UID: "9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:17:16.140326 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.140294 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" event={"ID":"6b6742db-d55d-4aac-b612-dafe9c875cc5","Type":"ContainerStarted","Data":"6e572a6a8a7b0d1be85b3ea7439df8113ba50f09f06e579055afa8891ed34fe4"} Nov 03 17:17:16.140768 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.140576 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:16.140768 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.140652 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:16.142777 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.142752 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mxk9" event={"ID":"b7b90722-afde-44ca-88ac-a73e3b3c61aa","Type":"ContainerStarted","Data":"ecd35d436f019f449c4ab84ee2d9650f18fed82070aa1baa5a19d7bca7889c6c"} Nov 03 17:17:16.142886 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.142800 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8mxk9" event={"ID":"b7b90722-afde-44ca-88ac-a73e3b3c61aa","Type":"ContainerStarted","Data":"dfbe1e29bf88b9ea5e4ef0383659ef7795df1d8c462cc436b1a1ee947aed4b7c"} Nov 03 17:17:16.156834 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.156815 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:16.389309 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.389255 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" podStartSLOduration=6.190313691 podStartE2EDuration="16.389235402s" totalImagesPullingTime="10.198921711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:16.362788825 +0000 UTC m=+38.848401705" watchObservedRunningTime="2026-11-03 17:17:16.389235402 +0000 UTC m=+38.874848283" Nov 03 17:17:16.624738 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:16.624684 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8mxk9" podStartSLOduration=3.127536215 podStartE2EDuration="4.624670362s" totalImagesPullingTime="1.421861609s" totalInitContainerRuntime="75.272538ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:16.624319918 +0000 UTC m=+39.109932796" watchObservedRunningTime="2026-11-03 17:17:16.624670362 +0000 UTC m=+39.110283240" Nov 03 17:17:17.146044 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.145584 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:17.160787 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.160763 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:17.946641 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.946610 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z526b"] Nov 03 17:17:17.946821 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.946755 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:17.946860 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:17.946829 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:17.949904 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.949879 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j65bb"] Nov 03 17:17:17.950013 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.949972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:17.950058 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:17.950042 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:17.950396 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.950377 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l54gp"] Nov 03 17:17:17.950481 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:17.950470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:17.950558 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:17.950544 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:19.986006 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:19.985966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:19.986892 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:19.986128 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:19.986892 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:19.986395 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:19.986892 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:19.986509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:19.987046 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:19.987012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:19.987149 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:19.987126 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:21.988671 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:21.988638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:21.989123 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:21.988718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:21.989123 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:21.988744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:21.989123 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:21.988830 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l54gp" podUID="266c9430-4b51-4065-a2fa-00f638011a0f" Nov 03 17:17:21.989123 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:21.988864 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j65bb" podUID="83f65e07-2574-44d8-9fe0-de9b7f689bd4" Nov 03 17:17:21.989123 ip-10-0-129-76 kubenswrapper[2578]: E1103 17:17:21.988953 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z526b" podUID="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" Nov 03 17:17:23.293594 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.293430 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeReady" Nov 03 17:17:23.294036 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.293745 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 17:17:23.463299 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.463220 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jmf7n"] Nov 03 17:17:23.485997 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.485965 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89c8l"] Nov 03 17:17:23.486194 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.486124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.496745 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.496726 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6rgjt"] Nov 03 17:17:23.496848 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.496833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.505901 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.505876 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmf7n"] Nov 03 17:17:23.506005 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.505988 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.556405 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.556379 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:17:23.557347 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.557326 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89c8l"] Nov 03 17:17:23.557855 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.557839 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6rgjt"] Nov 03 17:17:23.621353 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621285 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvczl\" (UniqueName: \"kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.621501 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.621501 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57g6\" (UniqueName: \"kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.621566 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.621598 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621570 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.621639 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621608 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.621684 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.621727 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.621727 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ng6\" (UniqueName: \"kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.621820 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.621820 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.621743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.626262 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.626245 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:17:23.672536 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.672494 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:17:23.672536 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.672507 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrssc\"" Nov 03 17:17:23.672938 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.672920 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gnkq2\"" Nov 03 17:17:23.673019 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.672931 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qk7gm\"" Nov 03 17:17:23.673019 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.672965 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:17:23.673019 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.673003 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:17:23.673019 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.673002 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:17:23.673209 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.673090 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:17:23.673209 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.673188 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:17:23.673287 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.673214 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:17:23.722874 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.722874 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.722874 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.723160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ng6\" (UniqueName: \"kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.723160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.722951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723160 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvczl\" (UniqueName: \"kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.723395 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.723395 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723395 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57g6\" (UniqueName: \"kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723395 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723611 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723473 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723611 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.723831 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723812 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.723970 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.723949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.726927 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.726905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.726927 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.726916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.727084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.726948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.789940 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.789910 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ng6\" (UniqueName: \"kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.790117 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.790040 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57g6\" (UniqueName: \"kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.790162 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.790152 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvczl\" (UniqueName: \"kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.811109 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.811045 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:23.831788 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.831744 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:23.839122 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.839094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:23.985908 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.985825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:23.986041 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.985893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:23.986041 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:23.985834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:24.042322 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.042297 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:17:24.042469 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.042345 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9v6p\"" Nov 03 17:17:24.042469 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.042435 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mr6xr\"" Nov 03 17:17:24.043430 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.043418 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d8tp8\"" Nov 03 17:17:24.043490 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.043460 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:17:24.043490 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.043484 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:17:24.043930 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.043919 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:17:24.236810 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:24.236759 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936ced82_e99e_4fee_8690_70ea861d6890.slice/crio-e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0 WatchSource:0}: Error finding container e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0: Status 404 returned error can't find the container with id e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0 Nov 03 17:17:24.238796 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.238778 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmf7n"] Nov 03 17:17:24.283415 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:24.283386 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e87cb76_bc36_418c_b7fd_393ccbe2d8db.slice/crio-77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3 WatchSource:0}: Error finding container 77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3: Status 404 returned error can't find the container with id 77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3 Nov 03 17:17:24.285523 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.285503 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6rgjt"] Nov 03 17:17:24.303683 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:24.303658 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eec4786_068b_439e_90cd_328eea55a1af.slice/crio-2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79 WatchSource:0}: Error finding container 2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79: Status 404 returned error can't find the container with id 2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79 Nov 03 17:17:24.305993 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.305975 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89c8l"] Nov 03 17:17:24.359199 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.359039 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:17:24.376914 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.376890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.420856 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.420821 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:17:24.487342 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.487208 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 17:17:24.487342 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.487280 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 17:17:24.487342 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.487325 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 17:17:24.487575 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.487387 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 17:17:24.487575 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.487441 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 17:17:24.532990 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.532949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.532990 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.532992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533215 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533215 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533215 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533325 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533325 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.533325 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.533279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.633936 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.633902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634114 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.633961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634114 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634114 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634114 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634114 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634355 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634355 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634444 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634782 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634762 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634823 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.634855 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.634809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.643616 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.642514 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 17:17:24.643616 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.642712 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9495t\"" Nov 03 17:17:24.643811 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.643704 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 17:17:24.645047 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.645020 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 17:17:24.646494 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.646406 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.648017 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.647996 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.648194 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.648178 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.699828 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.699798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:24.998705 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:24.998669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:25.165961 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:25.165927 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89c8l" event={"ID":"6eec4786-068b-439e-90cd-328eea55a1af","Type":"ContainerStarted","Data":"5b1aa474f9d2bbdc0d521db0103819d0e07a078f72c792eab5c4ce67de57ffd1"} Nov 03 17:17:25.166150 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:25.166008 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89c8l" event={"ID":"6eec4786-068b-439e-90cd-328eea55a1af","Type":"ContainerStarted","Data":"2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79"} Nov 03 17:17:25.168679 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:25.168643 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6rgjt" event={"ID":"6e87cb76-bc36-418c-b7fd-393ccbe2d8db","Type":"ContainerStarted","Data":"77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3"} Nov 03 17:17:25.170112 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:25.170061 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmf7n" event={"ID":"936ced82-e99e-4fee-8690-70ea861d6890","Type":"ContainerStarted","Data":"e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0"} Nov 03 17:17:25.405425 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:25.405347 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:17:26.175704 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:26.175654 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89c8l" event={"ID":"6eec4786-068b-439e-90cd-328eea55a1af","Type":"ContainerStarted","Data":"7d88da21be524f83e2e23660614ce68abe9c6298749082943d0b4ff094a3d59b"} Nov 03 17:17:26.177163 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:26.177131 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbf6c6fc9-tqhhc" event={"ID":"0c54a443-2e3f-4b1c-82b5-340188cae67a","Type":"ContainerStarted","Data":"b5f82a399b329e3eb8ab09f0684b063453c34f27b26426dc76aa1d5345f1af18"} Nov 03 17:17:27.184231 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:27.184199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmf7n" event={"ID":"936ced82-e99e-4fee-8690-70ea861d6890","Type":"ContainerStarted","Data":"7c1082a94521af2c80ffe6c5640056a8fc1a5b7546d73ebd53eed57b14818ba5"} Nov 03 17:17:28.135089 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.133603 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:28.135089 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.133760 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:17:28.135089 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.134189 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:28.192445 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.192364 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6rgjt" event={"ID":"6e87cb76-bc36-418c-b7fd-393ccbe2d8db","Type":"ContainerStarted","Data":"2728e759280c83a5093e9822d2c390c86ce0a08450e7df4e6a82e923e95004ce"} Nov 03 17:17:28.195998 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.195966 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmf7n" event={"ID":"936ced82-e99e-4fee-8690-70ea861d6890","Type":"ContainerStarted","Data":"5916ac2dd73fde460de71f6cb8220e93d95e00ce9b922dd2f215627fb0c4e9c5"} Nov 03 17:17:28.196171 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.196017 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:28.508327 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.508268 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jmf7n" podStartSLOduration=2.703776206 podStartE2EDuration="5.508253674s" totalImagesPullingTime="2.804477468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:28.508050872 +0000 UTC m=+50.993663751" watchObservedRunningTime="2026-11-03 17:17:28.508253674 +0000 UTC m=+50.993866552" Nov 03 17:17:28.508725 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:28.508697 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6rgjt" podStartSLOduration=1.776109247 podStartE2EDuration="5.508684136s" totalImagesPullingTime="3.732574889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:28.382992646 +0000 UTC m=+50.868605525" watchObservedRunningTime="2026-11-03 17:17:28.508684136 +0000 UTC m=+50.994297017" Nov 03 17:17:29.200239 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:29.200152 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89c8l" event={"ID":"6eec4786-068b-439e-90cd-328eea55a1af","Type":"ContainerStarted","Data":"a47907ccf397bddf20fa38080a851d0e7525c6dedf2076e4b99b13a432bfc672"} Nov 03 17:17:29.324709 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:29.324644 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89c8l" podStartSLOduration=1.905976538 podStartE2EDuration="6.324625574s" totalImagesPullingTime="4.418649036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:17:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:29.324286676 +0000 UTC m=+51.809899555" watchObservedRunningTime="2026-11-03 17:17:29.324625574 +0000 UTC m=+51.810238455" Nov 03 17:17:30.207136 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:30.207101 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbf6c6fc9-tqhhc" event={"ID":"0c54a443-2e3f-4b1c-82b5-340188cae67a","Type":"ContainerStarted","Data":"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49"} Nov 03 17:17:30.332938 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:30.332885 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fbf6c6fc9-tqhhc" podStartSLOduration=2.519643146 podStartE2EDuration="6.332872538s" totalImagesPullingTime="3.813229392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:30.331837484 +0000 UTC m=+52.817450364" watchObservedRunningTime="2026-11-03 17:17:30.332872538 +0000 UTC m=+52.818485416" Nov 03 17:17:31.996699 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:31.996663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:31.999717 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:31.999694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:32.097284 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.097241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:32.099630 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.099608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:32.126757 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.126722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:32.138959 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.138927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:32.510044 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.510002 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l54gp"] Nov 03 17:17:32.530553 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:32.530527 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266c9430_4b51_4065_a2fa_00f638011a0f.slice/crio-835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f WatchSource:0}: Error finding container 835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f: Status 404 returned error can't find the container with id 835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f Nov 03 17:17:32.547684 ip-10-0-129-76 kubenswrapper[2578]: W1103 17:17:32.547651 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb96298_6581_4c6b_aa6f_4a6ce4a07eeb.slice/crio-d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1 WatchSource:0}: Error finding container d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1: Status 404 returned error can't find the container with id d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1 Nov 03 17:17:32.550338 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:32.550318 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z526b"] Nov 03 17:17:33.218920 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:33.218881 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z526b" event={"ID":"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb","Type":"ContainerStarted","Data":"d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1"} Nov 03 17:17:33.219969 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:33.219943 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l54gp" event={"ID":"266c9430-4b51-4065-a2fa-00f638011a0f","Type":"ContainerStarted","Data":"835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f"} Nov 03 17:17:34.226367 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:34.226322 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l54gp" event={"ID":"266c9430-4b51-4065-a2fa-00f638011a0f","Type":"ContainerStarted","Data":"56b7f8513f5c1b4b976448418712c1c0179ad8573ac591e84d6b37fa1b17c96c"} Nov 03 17:17:34.226367 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:34.226375 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l54gp" event={"ID":"266c9430-4b51-4065-a2fa-00f638011a0f","Type":"ContainerStarted","Data":"46428fff4ec686b75b54ac5adb6a7894ed67f448446035f8ef6a5164d6dde27a"} Nov 03 17:17:34.346373 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:34.346274 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l54gp" podStartSLOduration=32.873106196 podStartE2EDuration="34.346256495s" totalImagesPullingTime="1.473150299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:34.34565783 +0000 UTC m=+56.831270709" watchObservedRunningTime="2026-11-03 17:17:34.346256495 +0000 UTC m=+56.831869376" Nov 03 17:17:34.999293 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:34.999258 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:34.999469 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:34.999367 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:35.005554 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:35.005527 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:35.234084 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:35.234038 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:37.468903 ip-10-0-129-76 kubenswrapper[2578]: I1103 17:17:37.468800 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:17:37.468946 ip-10-0-129-76 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 17:17:37.491455 ip-10-0-129-76 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 17:17:37.491673 ip-10-0-129-76 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 17:17:37.491869 ip-10-0-129-76 systemd[1]: kubelet.service: Consumed 3.115s CPU time. Nov 03 17:17:37.502739 ip-10-0-129-76 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:17:37.573560 ip-10-0-129-76 kubenswrapper[4885]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:17:37.573560 ip-10-0-129-76 kubenswrapper[4885]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:17:37.573560 ip-10-0-129-76 kubenswrapper[4885]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:17:37.573560 ip-10-0-129-76 kubenswrapper[4885]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:17:37.575671 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575595 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.575671 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575672 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575676 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575679 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575682 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575686 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575691 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575695 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575699 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575702 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575704 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575708 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575710 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575713 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575716 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575718 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575721 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575724 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575726 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575730 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.575738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575739 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575743 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575747 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575749 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575752 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575755 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575757 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575760 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575763 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575765 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575768 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575770 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575772 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575775 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575777 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575780 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575782 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575784 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575787 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575790 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.576266 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575792 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575795 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575797 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575801 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575803 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575805 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575808 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575811 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575813 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575817 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575819 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575822 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575824 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575827 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575829 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575832 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575834 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575837 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575839 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575842 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.576762 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575844 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575846 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575849 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575851 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575854 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575856 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575860 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575863 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575866 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575868 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575871 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575873 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575877 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575880 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575882 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575885 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575887 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575890 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575892 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575895 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.577278 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575897 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575900 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575903 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575905 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575908 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575911 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575913 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575916 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575918 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575921 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575923 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575926 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575928 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575931 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575933 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575936 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575938 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575941 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575943 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575946 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.577768 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575949 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575952 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575954 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575957 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575959 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.575963 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576751 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576759 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576763 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576766 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576768 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576771 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576774 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576777 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576780 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576783 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576786 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576788 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576791 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576794 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.578275 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576796 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576799 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576802 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576805 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576808 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576810 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576813 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576815 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576818 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576820 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576823 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576826 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576828 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576831 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576834 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576836 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576838 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576841 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576850 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.578778 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576852 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576855 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576857 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576860 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576863 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576865 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576868 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576871 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576874 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576876 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576879 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576881 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576884 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576886 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576889 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576891 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576894 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576896 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576899 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576901 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.579267 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576905 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576908 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576911 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576913 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576916 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576919 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576921 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576924 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576926 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576929 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576932 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576934 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576942 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576946 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576949 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576952 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576954 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576957 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576959 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.579764 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576962 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576964 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576967 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576969 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576972 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576974 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576977 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576979 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576982 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576984 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576986 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576989 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.576992 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577000 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577003 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577005 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577008 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577010 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577016 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.580252 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577019 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577023 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577028 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577031 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577034 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577037 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577039 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577047 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577051 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577053 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577056 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577058 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577061 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577078 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577082 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577581 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577587 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577590 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577592 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.580841 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577595 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577598 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577600 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577603 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577605 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577608 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577611 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577613 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577616 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577618 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577621 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577624 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577627 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577629 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577632 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577635 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577638 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577641 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577643 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577646 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.581400 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577651 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577654 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577657 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577659 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577662 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577664 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577667 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577669 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577671 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577674 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577676 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577680 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577683 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577685 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577688 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577690 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577693 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577696 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577699 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577701 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.582282 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577704 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577706 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577709 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577711 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577714 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577716 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577719 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577721 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577724 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577726 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577728 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577731 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577733 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577736 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577739 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577743 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577746 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577749 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577752 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.583198 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577755 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577758 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577760 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577762 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577765 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577767 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577770 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577772 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577776 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577779 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577781 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577784 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577786 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577789 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577791 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577794 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577796 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577799 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577802 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577804 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.584026 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577807 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577809 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577812 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577814 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577817 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577819 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577824 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577828 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577832 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577835 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577837 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577840 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577842 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577845 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577847 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577850 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577853 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577856 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577858 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.584964 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577861 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577866 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577868 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.577871 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.577961 4885 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.577968 4885 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.577984 4885 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.577991 4885 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.577998 4885 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578002 4885 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578007 4885 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578011 4885 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578014 4885 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578024 4885 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578027 4885 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578031 4885 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578034 4885 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578037 4885 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578039 4885 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578042 4885 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578045 4885 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578049 4885 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578055 4885 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578058 4885 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:17:37.585783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578061 4885 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578077 4885 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578080 4885 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578088 4885 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578091 4885 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578094 4885 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578097 4885 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578100 4885 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578103 4885 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578107 4885 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578110 4885 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578113 4885 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578117 4885 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578120 4885 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578122 4885 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578125 4885 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578128 4885 flags.go:64] FLAG: --enable-server="true" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578131 4885 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578136 4885 flags.go:64] FLAG: --event-burst="100" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578139 4885 flags.go:64] FLAG: --event-qps="50" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578142 4885 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578145 4885 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578147 4885 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578151 4885 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:17:37.586571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578155 4885 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578157 4885 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578160 4885 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578163 4885 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578166 4885 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578169 4885 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578173 4885 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578175 4885 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578178 4885 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578181 4885 flags.go:64] FLAG: --feature-gates="" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578185 4885 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578189 4885 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578192 4885 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578196 4885 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578198 4885 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578201 4885 flags.go:64] FLAG: --help="false" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578204 4885 flags.go:64] FLAG: --hostname-override="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578208 4885 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578211 4885 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578214 4885 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578218 4885 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578221 4885 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578224 4885 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:17:37.587235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578227 4885 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578230 4885 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578233 4885 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578236 4885 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578239 4885 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578242 4885 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578245 4885 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578247 4885 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578250 4885 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578253 4885 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578256 4885 flags.go:64] FLAG: --lock-file="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578258 4885 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578261 4885 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578264 4885 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578269 4885 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578272 4885 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578276 4885 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578279 4885 flags.go:64] FLAG: --logging-format="text" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578282 4885 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578286 4885 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578290 4885 flags.go:64] FLAG: --manifest-url="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578293 4885 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578297 4885 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578300 4885 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578304 4885 flags.go:64] FLAG: --max-pods="110" Nov 03 17:17:37.587820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578307 4885 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578311 4885 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578314 4885 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578317 4885 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578320 4885 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578323 4885 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578326 4885 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578333 4885 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578336 4885 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578339 4885 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578342 4885 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578344 4885 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578347 4885 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578350 4885 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578353 4885 flags.go:64] FLAG: --port="10250" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578356 4885 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578359 4885 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c830310aab35c948" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578362 4885 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578364 4885 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578367 4885 flags.go:64] FLAG: --register-node="true" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578370 4885 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578374 4885 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578377 4885 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578380 4885 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578383 4885 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578387 4885 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:17:37.588509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578390 4885 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578394 4885 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578397 4885 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578400 4885 flags.go:64] FLAG: --runonce="false" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578403 4885 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578406 4885 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578409 4885 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578413 4885 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578415 4885 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578418 4885 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578422 4885 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578424 4885 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578427 4885 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578430 4885 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578433 4885 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578435 4885 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578438 4885 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578441 4885 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578444 4885 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578449 4885 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578452 4885 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578455 4885 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578458 4885 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578461 4885 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578464 4885 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:17:37.589157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578467 4885 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578469 4885 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578472 4885 flags.go:64] FLAG: --v="2" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578476 4885 flags.go:64] FLAG: --version="false" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578484 4885 flags.go:64] FLAG: --vmodule="" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578488 4885 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578492 4885 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578629 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578635 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578638 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578640 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578643 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578645 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578648 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578652 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578654 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578657 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578660 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578662 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578665 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578667 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.589797 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578669 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578672 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578674 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578677 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578679 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578682 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578685 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578687 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578690 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578692 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578695 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578697 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578700 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578702 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578705 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578707 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578710 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578713 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578716 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578720 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.590524 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578723 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578725 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578728 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578730 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578733 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578737 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578739 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578742 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578744 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578747 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578749 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578753 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578757 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578760 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578763 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578766 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578769 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578771 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578774 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.591299 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578777 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578779 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578782 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578785 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578789 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578793 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578795 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578798 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578800 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578803 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578805 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578808 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578813 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578816 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578818 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578821 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578824 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578826 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578830 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.591951 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578833 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578835 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578838 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578841 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578843 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578846 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578848 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578850 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578853 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578855 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578858 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578861 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578863 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578866 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578868 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578871 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578873 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578875 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578878 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578880 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.592518 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578883 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578885 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578888 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578890 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578893 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578897 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578901 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578903 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578906 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578908 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578911 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578915 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578918 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578920 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.578947 4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578981 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.593005 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578985 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578988 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578991 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578994 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578996 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.578999 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579002 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579004 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579007 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579009 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579012 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579014 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579017 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579019 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579022 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579024 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579026 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579029 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579032 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579034 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.593458 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579037 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579040 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579043 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579047 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579049 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579052 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579054 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579058 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579061 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579076 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579079 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579081 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579084 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579087 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579090 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579092 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579095 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579099 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579102 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579104 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.594008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579107 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579110 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579112 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579114 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579117 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579119 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579122 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579124 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579128 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579131 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579134 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579137 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579139 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579143 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579147 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579150 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579153 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579156 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579159 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.594554 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579163 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579167 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579171 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579174 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579177 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579180 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579182 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579185 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579187 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579190 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579192 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579195 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579197 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579200 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579202 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579205 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579207 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579210 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579212 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.595021 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579215 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579218 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579220 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579223 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579225 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579228 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579230 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579233 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579237 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579239 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579242 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579245 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579247 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579252 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579255 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579257 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579260 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579262 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579265 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579267 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.595576 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579270 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579273 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579275 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579279 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579282 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579284 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.579287 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.584575 4885 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.584596 4885 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.584620 4885 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.584628 4885 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584723 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584731 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584736 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584740 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584744 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584749 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584753 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584757 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.596098 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584762 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584768 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584774 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584779 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584784 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584788 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584792 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584796 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584803 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584809 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584814 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584819 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584823 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584828 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584832 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584836 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584840 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584844 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584849 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.596572 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584855 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584859 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.584926 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585191 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585203 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585208 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585213 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585217 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585222 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585226 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585231 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585236 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585240 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585245 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585250 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585255 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585265 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585270 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585274 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585279 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.597025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585283 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585288 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585293 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585297 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585302 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585306 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585311 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585315 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585319 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585329 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585334 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585338 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585343 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585347 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585352 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585390 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585395 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585400 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585404 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585409 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.597543 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585414 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585424 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585428 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585433 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585437 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585441 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585445 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585450 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585455 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585460 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585464 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585469 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585473 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585477 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585487 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585492 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585497 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585501 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585505 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585510 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.598025 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585514 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585519 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585526 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585531 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585535 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585542 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585546 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585557 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585592 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585599 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585605 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585610 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585616 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585621 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585626 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585632 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585636 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585641 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.598539 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.585645 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.585709 4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586389 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586401 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586404 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586408 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586411 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586413 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586417 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586419 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586422 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586425 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586427 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586430 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586433 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586435 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.599008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586438 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586440 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586443 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586445 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586448 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586452 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586456 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586460 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586463 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586466 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586469 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586471 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586474 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586476 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586479 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586482 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586485 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586487 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586490 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.599415 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586493 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586496 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586498 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586501 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586504 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586506 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586509 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586512 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586514 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586517 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586520 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586522 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586525 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586527 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586530 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586533 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586536 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586539 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586542 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586545 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.599880 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586547 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586550 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586552 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586556 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586558 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586561 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586563 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586565 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586568 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586571 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586573 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586576 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586579 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586582 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586585 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586587 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586591 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586593 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586597 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586599 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.600384 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586602 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586605 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586607 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586610 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586612 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586614 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586617 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586620 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586622 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586625 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586627 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586630 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586633 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586635 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586638 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586641 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586645 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586648 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586650 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.600882 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586653 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586656 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586658 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586661 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586663 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586667 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586669 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586672 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586674 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586677 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586679 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586682 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586685 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586688 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.586718 4885 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586753 4885 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:17:37.601355 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586758 4885 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586761 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586764 4885 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586767 4885 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586770 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586773 4885 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586775 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586778 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586780 4885 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586783 4885 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586785 4885 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586788 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586790 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586793 4885 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586795 4885 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586798 4885 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586800 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586803 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586805 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586808 4885 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:17:37.601738 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586810 4885 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586813 4885 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586816 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586818 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586821 4885 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586823 4885 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586826 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586828 4885 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586831 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586834 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586837 4885 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586839 4885 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586842 4885 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586844 4885 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586846 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586849 4885 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586852 4885 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586854 4885 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586857 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586859 4885 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:17:37.602308 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586863 4885 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586868 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586870 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586873 4885 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586876 4885 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586879 4885 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586881 4885 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586884 4885 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586887 4885 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586889 4885 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586891 4885 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586894 4885 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586896 4885 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586899 4885 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586902 4885 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586905 4885 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586907 4885 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586910 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586913 4885 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:17:37.602774 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586916 4885 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586918 4885 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586921 4885 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586924 4885 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586926 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586929 4885 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586931 4885 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586934 4885 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586936 4885 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586938 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586941 4885 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586943 4885 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586946 4885 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586948 4885 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586951 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586953 4885 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586956 4885 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586959 4885 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586963 4885 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:17:37.603410 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586966 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586968 4885 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586971 4885 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586973 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586976 4885 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586979 4885 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586983 4885 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586986 4885 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586989 4885 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586991 4885 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586994 4885 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.586997 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587001 4885 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587003 4885 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587006 4885 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587008 4885 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587011 4885 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587014 4885 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587017 4885 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587019 4885 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:17:37.604143 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587022 4885 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587024 4885 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587027 4885 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587029 4885 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587032 4885 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587035 4885 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:37.587037 4885 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.587106 4885 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588055 4885 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588128 4885 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588309 4885 server.go:1014] "Starting client certificate rotation" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588451 4885 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588518 4885 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:11:37 +0000 UTC" deadline="2028-06-29 11:44:48.65607161 +0000 UTC" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588557 4885 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14490h27m11.067517913s" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.588848 4885 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.589947 4885 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:17:37.604859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.591594 4885 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:17:37.605358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.593321 4885 log.go:25] "Validated CRI v1 image API" Nov 03 17:17:37.605358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.593988 4885 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:17:37.605358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.595599 4885 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d7987b8b-663a-463c-a9d0-ed8301acd187:/dev/nvme0n1p3 f0a42282-1e06-4d37-829b-4383f2931079:/dev/nvme0n1p4] Nov 03 17:17:37.605494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.595613 4885 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076/userdata/shm major:0 minor:66 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99/userdata/shm major:0 minor:132 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298/userdata/shm major:0 minor:128 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f/userdata/shm major:0 minor:282 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873/userdata/shm major:0 minor:192 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5f82a399b329e3eb8ab09f0684b063453c34f27b26426dc76aa1d5345f1af18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5f82a399b329e3eb8ab09f0684b063453c34f27b26426dc76aa1d5345f1af18/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172/userdata/shm major:0 minor:58 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b/userdata/shm major:0 minor:114 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0/userdata/shm major:0 minor:115 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~projected/kube-api-access-br2d5:{mountpoint:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~projected/kube-api-access-br2d5 major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~projected/kube-api-access-t7b5m:{mountpoint:/var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~projected/kube-api-access-t7b5m major:0 minor:108 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d6141b3-1493-4bd1-b637-5d4290f46930/volumes/kubernetes.io~projected/kube-api-access-hn2bl:{mountpoint:/var/lib/kubelet/pods/2d6141b3-1493-4bd1-b637-5d4290f46930/volumes/kubernetes.io~projected/kube-api-access-hn2bl major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37c806b4-ceab-4062-aa31-e0776a71ed05/volumes/kubernetes.io~projected/kube-api-access-zlvtx:{mountpoint:/var/lib/kubelet/pods/37c806b4-ceab-4062-aa31-e0776a71ed05/volumes/kubernetes.io~projected/kube-api-access-zlvtx major:0 minor:101 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54863910-20c1-44fe-b7f7-f2a39321a9eb/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/54863910-20c1-44fe-b7f7-f2a39321a9eb/volumes/kubernetes.io~secret/agent-certs major:0 minor:112 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~projected/kube-api-access-md5n4:{mountpoint:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~projected/kube-api-access-md5n4 major:0 minor:109 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:102 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~projected/kube-api-access-nvczl:{mountpoint:/var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~projected/kube-api-access-nvczl major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~secret/cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~projected/kube-api-access-v57g6:{mountpoint:/var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~projected/kube-api-access-v57g6 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~projected/kube-api-access-d6ng6:{mountpoint:/var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~projected/kube-api-access-d6ng6 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~secret/metrics-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b53025bb-d955-4dd9-a75e-74ad4bdaf61b/volumes/kubernetes.io~projected/kube-api-access-tbqhl:{mountpoint:/var/lib/kubelet/pods/b53025bb-d955-4dd9-a75e-74ad4bdaf61b/volumes/kubernetes.io~projected/kube-api-access-tbqhl major:0 minor:107 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~projected/kube-api-access-sczxl:{mountpoint:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~projected/kube-api-access-sczxl major:0 minor:191 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:186 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:190 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb8b9a87-1619-4d2c-8989-3bba620ce5b5/volumes/kubernetes.io~projected/kube-api-access-2hvsv:{mountpoint:/var/lib/kubelet/pods/bb8b9a87-1619-4d2c-8989-3bba620ce5b5/volumes/kubernetes.io~projected/kube-api-access-2hvsv major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:77 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/tmp major:0 minor:76 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~projected/kube-api-access-vqgfm:{mountpoint:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~projected/kube-api-access-vqgfm major:0 minor:80 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e35b8f38-9b01-4c09-85d8-560e93cd71da/volumes/kubernetes.io~projected/kube-api-access-2x6zw:{mountpoint:/var/lib/kubelet/pods/e35b8f38-9b01-4c09-85d8-560e93cd71da/volumes/kubernetes.io~projected/kube-api-access-2x6zw major:0 minor:113 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f39eb672-c5af-4f63-b7ef-b34fa3674c6d/volumes/kubernetes.io~projected/kube-api-access-bjp9n:{mountpoint:/var/lib/kubelet/pods/f39eb672-c5af-4f63-b7ef-b34fa3674c6d/volumes/kubernetes.io~projected/kube-api-access-bjp9n major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/cab2a4cd62d89bad52c29a4259a50df6941fb02f163d22a357fda9d6d739402c/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/755f6b768a2373b573ac86be4ba29cc2fd08f8d4e10408bc63bf1ea931d7b149/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/containers/storage/overlay/4d0f842d6ac74f41e7f2415c19b126d1ea0ba6dec6e76a0281d2b9c99015e277/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-120:{mountpoint:/var/lib/containers/storage/overlay/5eec251a0a58bb08338e560949f3b73a5692cc66f10ee091c045f6c961462897/merged major:0 minor:120 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/23cfef734b3a72415ed1326a42c8fa250289664344e8f05c7f651ca2b2359ef1/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a978fc3e58806ec8eec7d984f3adf88c2a4d77a9eff5a6eb578e84078bcf163/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/0c0863cea81af2dc5e66966b03b96d498aae7d86e9246692df6b7cee6376ed61/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/dc950bd41233846f145cfdbad90e78552cbb587a797fce6c8492779dfa854a2c/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e70f9a8456614ffa31a96b254964b8d7eefbc2abfa7f3dfb6d3a32a6edaf330a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/00dc197913f93d9a7d272deee0576c7c56c6db6fd59b3c1ad04a742484eae1e4/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4dd861d99f10b10c35c292e511f6df178e6baaf6a9c6db4d9470659ce98f9c5c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/a547bf0780dd4ca51f2e1d97b2f854b340ff0899c66e93082e6069493696627a/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/765944055d469994f6ff7653ee60d30246073f0786f891ab42b587d18fd9d48e/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/58119f2dbc47f6598aa2b46946124e35470cd48fda2f2d7f37570a826eea2bc6/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/9ac0db945afa87a37f8cfa367cc2fdb34784ad70804e8ddc3c5793240e1e3f13/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/30223d17f2bca9e7aaff91fb42cfed35aa14b6464de28c76e4d769db7788765e/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/5c40800fff93849d64c6b1b23a2b2e896401a05525a080b643abc02d2b17f9bb/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/efc404c8a4eb19a4e7946641d2795f29db044b700548b807234d1748c8ca80f1/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-196:{mountpoint:/var/lib/containers/storage/overlay/fad338d55c0a99f2d614b76489b75a532d4fc118e444dfce4b7a215b20d705ef/merged major:0 minor:196 fsType:overlay blockSize:0} overlay_0-198:{mountpoint:/var/lib/containers/storage/overlay/623df7e009c84079edf1f685cb4a314075f18882e9f5d61153fe9dfa317e3152/merged major:0 minor:198 fsType:overlay blockSize:0} overlay_0-203:{mountpoint:/var/lib/containers/storage/overlay/b075a8991373f597ea0b83e883b77f57e2ba50d4f0046975c2cd0c619d211d58/merged major:0 minor:203 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/71c12dfb7a60f07e6cd71efd0078d2f92c0a83e46c96ec514985b3b00144fd13/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/a18cd338ebfa718b8ae7c5ac4454f6862c4c4510a60ec0634654a151fd5dd664/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4ee4dca7b05279cac2d8bd7439dd7b0f4bb4a9401109b4265fb2cdc7f1536247/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/901ab50f141d8bda81a900596d639b5556b536567344bd7df5e50458d8e8ff6e/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/c746794b56b7b7ace33f9d942263fcda65f5069705e48959f0be23e25b2694a1/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/0da55a9f987c9ec2fd6995a06c66034bff1c2e3dc2c806d992b0ca6b7437d795/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/dcfeda32d2df70c8b7cd35d0693d7f1cbe442a642322525ee27b06b647b61017/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/009aa2f8d2d0234b8e6631a0306110e9a5079f8abc946328e68d25037acb1018/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/7a0ca33d349ce985dedd14d9956469de8d637ce6e7f2856f5bab7f5c64ecdbca/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c43ab608c54b6801e4f49730b15170497a6260e2e0934b35728d1bf32fab3d9a/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/f1d815f26f26bfcbbd3c164554d5678140a12730c3e080cf147c405dbdd4f927/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/1df51c624173cd958ea22e46ecaecc2e9f2756f670fe127e12c61375d8011b67/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9f25aa03e05969b4ec3319d4fbf860f7fa2d260a1a943c99233ab06deae38a4b/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/52bd6125dca17415cbbd75557fc9cc2fe9f5af1b20860d3552dee15bbf077d19/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/3e53abd0fc29c42659bacda277ee0d59c35d6bd36be907908cffb89444b100bf/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/7832d698f82a3d6c4f0401c15b5dc4bd966d775deb159de4726a57b67452c362/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0276dc3efb258663a92bae54970336adc7463db2d05eac9364cc5ec0062411b4/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/a8e025deb115442ddbd132d2db59a1082dbcdeb37bcd21ad393acf1f4940dcdd/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/eb447995f71f126d959bd9fadc3f63761642a33dc73ec0304e969ef589b0e941/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/2cccb2ef1d879f140c8250664d095ac0e1fc501ac6b3227c37f0d5a83ca062c0/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-60:{mountpoint:/var/lib/containers/storage/overlay/6e67c4dcc0b8308f436a3d03be3f3b0ce10a11f0a7fc5f86515c8541a49c6da3/merged major:0 minor:60 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/945701cd4963bff0601a0f586d9442c96ba7f296ba9319e2cf09d5120f1c51af/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/4330ce3d0335c9b2d98d22534a64c32f6452abc435e90e0ae4c4153dfd840010/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5d4b0e410744fc3985412541104ad795ffb0dc997f52a782d0649b6869e89b0b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f374bd946dd9627579806cc41350a3fe67495da782f01ed27eac781097959df9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/46d3e2dd054deaeb38a72394440d19606cd17ce9aac0aebaead97207e6ba09b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1134ad8826b01a5b51d7ba162a6b53628a59792ddae182b8ef9d776d18e1bb7c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/7408a59ff85d00e10cbea4f76e5d9982c39347438480a94d4107a23524591e91/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/1e9250f4710e5c5cdf7725366b83034d49b8415fcde63e355731410f7223694e/merged major:0 minor:89 fsType:overlay blockSize:0}] Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.603121 4885 manager.go:217] Machine: {Timestamp:2026-11-03 17:17:37.602411842 +0000 UTC m=+0.057100858 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099600 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c5737f093a362ad076d637691ddda SystemUUID:ec2c5737-f093-a362-ad07-6d637691ddda BootID:8d992db9-135f-4a60-99bb-a65ea56bf896 Filesystems:[{Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b53025bb-d955-4dd9-a75e-74ad4bdaf61b/volumes/kubernetes.io~projected/kube-api-access-tbqhl DeviceMajor:0 DeviceMinor:107 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~projected/kube-api-access-sczxl DeviceMajor:0 DeviceMinor:191 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:77 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7ea66d953a84e902f28eeff89ab99ed82f4c6a3e9db26bfa4fd1f9907ba22d0/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2264f048139f7f6fa178d6904a1087c1e1d788828113d2b5b7334ce27df62a1/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33fd0789156af22d15b5b6b71b7cf51a271b462be0669079fb93248a6c72894d/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:190 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-198 DeviceMajor:0 DeviceMinor:198 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72d5fcdd2bdcfad9956026161b98240c01edeb8fdce943350f6a0b56be55cb99/userdata/shm DeviceMajor:0 DeviceMinor:132 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2ff6aa86de4aa441d91ab7fa5599091970299f6cf9f20bf5f8d9d113c9725c9/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~projected/kube-api-access-t7b5m DeviceMajor:0 DeviceMinor:108 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~projected/kube-api-access-md5n4 DeviceMajor:0 DeviceMinor:109 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e24700d044fc2ef037fcbf075779fe23f1a05c5c3e9558cb3c1b58dbaf3442d0/userdata/shm DeviceMajor:0 DeviceMinor:115 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-120 DeviceMajor:0 DeviceMinor:120 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2d6141b3-1493-4bd1-b637-5d4290f46930/volumes/kubernetes.io~projected/kube-api-access-hn2bl DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50db692216a991cf3318b02578101187653575813fd20c3a3d417c607f4fe076/userdata/shm DeviceMajor:0 DeviceMinor:66 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a9eb2f6eabe72f9304ded3df695ee2e271d59c197614712ca00ac7b7a9a2298/userdata/shm DeviceMajor:0 DeviceMinor:128 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6eec4786-068b-439e-90cd-328eea55a1af/volumes/kubernetes.io~projected/kube-api-access-v57g6 DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f39eb672-c5af-4f63-b7ef-b34fa3674c6d/volumes/kubernetes.io~projected/kube-api-access-bjp9n DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caf6c2000466da2c9044cb8d9f9de88a620cb23a7a3ca94323911bd4d890b172/userdata/shm DeviceMajor:0 DeviceMinor:58 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7b90722-afde-44ca-88ac-a73e3b3c61aa/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:186 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f535aa79dc9ac2a4f0951832c8cf7ae75ba8d8a50969af60a17b548ba67a873/userdata/shm DeviceMajor:0 DeviceMinor:192 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/266c9430-4b51-4065-a2fa-00f638011a0f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:276 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~projected/kube-api-access-nvczl DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:281 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:76 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-203 DeviceMajor:0 DeviceMinor:203 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~projected/kube-api-access-br2d5 DeviceMajor:0 DeviceMinor:242 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2881b5952176fc7c5c4f296b2c809039849e3fd24f10fcf4fce2bdac975c8b79/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5f82a399b329e3eb8ab09f0684b063453c34f27b26426dc76aa1d5345f1af18/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/936ced82-e99e-4fee-8690-70ea861d6890/volumes/kubernetes.io~projected/kube-api-access-d6ng6 DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/bb8b9a87-1619-4d2c-8989-3bba620ce5b5/volumes/kubernetes.io~projected/kube-api-access-2hvsv DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-60 DeviceMajor:0 DeviceMinor:60 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0795d540709f0b106a03fe87ef916a2af1b53585e0a5d0550c7eb132de87f2b/userdata/shm DeviceMajor:0 DeviceMinor:114 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca30f271-2ab1-468f-8413-64c76f5eb172/volumes/kubernetes.io~projected/kube-api-access-vqgfm DeviceMajor:0 DeviceMinor:80 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37c806b4-ceab-4062-aa31-e0776a71ed05/volumes/kubernetes.io~projected/kube-api-access-zlvtx DeviceMajor:0 DeviceMinor:101 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54863910-20c1-44fe-b7f7-f2a39321a9eb/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:112 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/835aeff8be10b4e9bfc32a919874bcae0f9a6ebf71ba8387a9567539d1577e4f/userdata/shm DeviceMajor:0 DeviceMinor:282 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e35b8f38-9b01-4c09-85d8-560e93cd71da/volumes/kubernetes.io~projected/kube-api-access-2x6zw DeviceMajor:0 DeviceMinor:113 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-117 DeviceMajor:0 DeviceMinor:117 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-196 DeviceMajor:0 DeviceMinor:196 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e87cb76-bc36-418c-b7fd-393ccbe2d8db/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b6742db-d55d-4aac-b612-dafe9c875cc5/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:102 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77144c686ef877f9223d1d36abfd3ec81fa5d8dd7169fb4f7a02600d2d1fd2d3/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2881b5952176fc7 MacAddress:e6:11:06:92:9c:1e Speed:10000 Mtu:8901} {Name:77144c686ef877f MacAddress:22:d7:fc:cc:68:81 Speed:10000 Mtu:8901} {Name:835aeff8be10b4e MacAddress:1e:52:24:88:a5:47 Speed:10000 Mtu:8901} {Name:b5f82a399b329e3 MacAddress:16:c9:46:4d:1d:5f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f4:a0:ef:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:40:1e:23:bc:92 Speed:0 Mtu:8901} {Name:d2264f048139f7f MacAddress:12:8d:6b:81:f6:8c Speed:10000 Mtu:8901} {Name:e7ea66d953a84e9 MacAddress:fe:36:65:79:d1:67 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f4:a0:ef:9f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:2c:72:e8:3f:e6 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:db:e6:d4:32:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.603942 4885 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604037 4885 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604772 4885 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604793 4885 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-76.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604920 4885 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604928 4885 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604938 4885 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.604951 4885 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605029 4885 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605197 4885 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605296 4885 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605348 4885 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605361 4885 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605371 4885 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605380 4885 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:17:37.605762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605388 4885 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:17:37.606718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.605989 4885 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:17:37.606718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.606000 4885 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:17:37.606718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.606488 4885 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:17:37.608775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.608754 4885 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:17:37.608775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.608769 4885 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:17:37.608937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.608882 4885 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 17:17:37.609111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609096 4885 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:17:37.609179 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609129 4885 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:17:37.609264 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609250 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:17:37.609311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609276 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:17:37.609311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609289 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:17:37.609311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609302 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609316 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609328 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609338 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609343 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609350 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609355 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609372 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609383 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609414 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:17:37.609440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.609423 4885 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:17:37.612752 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.612739 4885 server.go:1263] "Started kubelet" Nov 03 17:17:37.614013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.613042 4885 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:17:37.614013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.613519 4885 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:17:37.614013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.613641 4885 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:17:37.613374 ip-10-0-129-76 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:17:37.614923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.614728 4885 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:17:37.617484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.616813 4885 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:17:37.617635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.617619 4885 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:17:37.618303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618280 4885 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:17:37.618431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618361 4885 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:17:37.618508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618500 4885 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:17:37.618704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618664 4885 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:11:59 +0000 UTC" deadline="2028-03-29 23:21:36.855136456 +0000 UTC" Nov 03 17:17:37.618802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618787 4885 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12294h3m59.236357748s" Nov 03 17:17:37.618877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.618372 4885 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:17:37.618962 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.618417 4885 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:17:37.620198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620179 4885 factory.go:153] Registering CRI-O factory Nov 03 17:17:37.620198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620201 4885 factory.go:223] Registration of the crio container factory successfully Nov 03 17:17:37.620346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620262 4885 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:17:37.620346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620272 4885 factory.go:55] Registering systemd factory Nov 03 17:17:37.620346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620280 4885 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:17:37.620346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620296 4885 factory.go:103] Registering Raw factory Nov 03 17:17:37.620346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620312 4885 manager.go:1199] Started watching for new ooms in manager Nov 03 17:17:37.620678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.620664 4885 manager.go:319] Starting recovery of all containers Nov 03 17:17:37.621339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621284 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6141b3-1493-4bd1-b637-5d4290f46930" volumeName="kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy" seLinuxMountContext="" Nov 03 17:17:37.621415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621344 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eec4786-068b-439e-90cd-328eea55a1af" volumeName="kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6" seLinuxMountContext="" Nov 03 17:17:37.621415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621359 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936ced82-e99e-4fee-8690-70ea861d6890" volumeName="kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir" seLinuxMountContext="" Nov 03 17:17:37.621415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621381 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53025bb-d955-4dd9-a75e-74ad4bdaf61b" volumeName="kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl" seLinuxMountContext="" Nov 03 17:17:37.621415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621393 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb8b9a87-1619-4d2c-8989-3bba620ce5b5" volumeName="kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv" seLinuxMountContext="" Nov 03 17:17:37.621415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621411 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621425 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6141b3-1493-4bd1-b637-5d4290f46930" volumeName="kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621438 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e87cb76-bc36-418c-b7fd-393ccbe2d8db" volumeName="kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621459 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eec4786-068b-439e-90cd-328eea55a1af" volumeName="kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621473 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eec4786-068b-439e-90cd-328eea55a1af" volumeName="kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621491 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936ced82-e99e-4fee-8690-70ea861d6890" volumeName="kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621506 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb" volumeName="kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621523 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb8b9a87-1619-4d2c-8989-3bba620ce5b5" volumeName="kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621536 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621562 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6141b3-1493-4bd1-b637-5d4290f46930" volumeName="kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621578 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54863910-20c1-44fe-b7f7-f2a39321a9eb" volumeName="kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621590 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6742db-d55d-4aac-b612-dafe9c875cc5" volumeName="kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621609 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6742db-d55d-4aac-b612-dafe9c875cc5" volumeName="kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621623 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6742db-d55d-4aac-b612-dafe9c875cc5" volumeName="kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621642 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936ced82-e99e-4fee-8690-70ea861d6890" volumeName="kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621655 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53025bb-d955-4dd9-a75e-74ad4bdaf61b" volumeName="kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config" seLinuxMountContext="" Nov 03 17:17:37.621662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621669 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621687 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621700 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6742db-d55d-4aac-b612-dafe9c875cc5" volumeName="kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621718 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621732 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621745 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30f271-2ab1-468f-8413-64c76f5eb172" volumeName="kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621762 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30f271-2ab1-468f-8413-64c76f5eb172" volumeName="kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621789 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e35b8f38-9b01-4c09-85d8-560e93cd71da" volumeName="kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621801 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621819 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b6742db-d55d-4aac-b612-dafe9c875cc5" volumeName="kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621832 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b53025bb-d955-4dd9-a75e-74ad4bdaf61b" volumeName="kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621845 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621866 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d6141b3-1493-4bd1-b637-5d4290f46930" volumeName="kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621880 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37c806b4-ceab-4062-aa31-e0776a71ed05" volumeName="kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621897 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621910 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621924 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f39eb672-c5af-4f63-b7ef-b34fa3674c6d" volumeName="kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621944 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621960 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621979 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54863910-20c1-44fe-b7f7-f2a39321a9eb" volumeName="kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.621996 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eec4786-068b-439e-90cd-328eea55a1af" volumeName="kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622011 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266c9430-4b51-4065-a2fa-00f638011a0f" volumeName="kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622032 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="266c9430-4b51-4065-a2fa-00f638011a0f" volumeName="kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622045 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37c806b4-ceab-4062-aa31-e0776a71ed05" volumeName="kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622082 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e87cb76-bc36-418c-b7fd-393ccbe2d8db" volumeName="kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622093 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="936ced82-e99e-4fee-8690-70ea861d6890" volumeName="kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622105 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7b90722-afde-44ca-88ac-a73e3b3c61aa" volumeName="kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622121 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca30f271-2ab1-468f-8413-64c76f5eb172" volumeName="kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622135 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e35b8f38-9b01-4c09-85d8-560e93cd71da" volumeName="kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622152 4885 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c54a443-2e3f-4b1c-82b5-340188cae67a" volumeName="kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert" seLinuxMountContext="" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622165 4885 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:17:37.622426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.622173 4885 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:17:37.653119 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.653053 4885 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:17:37.655009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.654778 4885 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:17:37.655009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.654804 4885 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:17:37.655009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.654826 4885 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:17:37.655009 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.654866 4885 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 17:17:37.668728 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668671 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="f67302a155cda0e970af4aeb435de933bd3b5b0ff0f22c9077d623703d01413e" exitCode=0 Nov 03 17:17:37.668728 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668726 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="35a746bd7e1638e0512cb0211e7468643abd2cf7f5fb6a2bde990c289187079e" exitCode=0 Nov 03 17:17:37.668923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668741 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="87d26756fdf80db61efe42d933e6d36484aae865e29bd6c9690ac5318cb1d4e5" exitCode=0 Nov 03 17:17:37.668923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668753 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="70c6a7cf4fc203052f18dbe1150a825a6856086adda4f4cdbaf6e5ef00816cee" exitCode=0 Nov 03 17:17:37.668923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668764 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="cc4706d9ca8c315dd047036685faec0eec1fc89a6d41f4e58000d88f67f8a739" exitCode=0 Nov 03 17:17:37.668923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.668773 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d6141b3-1493-4bd1-b637-5d4290f46930" containerID="a469eb15a493fb0399a22d31c9f9d3de6accf2a8ec527f0c0acc1f4724814860" exitCode=0 Nov 03 17:17:37.673626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.673597 4885 generic.go:356] "Generic (PLEG): container finished" podID="b7b90722-afde-44ca-88ac-a73e3b3c61aa" containerID="5ff053078609eab0f476adc6cfa64bfa5cc68b5af5bf4ded2b36e9c67bb398b4" exitCode=0 Nov 03 17:17:37.688148 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.688119 4885 generic.go:356] "Generic (PLEG): container finished" podID="bf9034c461ae30eb2c88e2ba1d28d459" containerID="3a7994358f7fee92a17f84628cf3a90399c28ac31a7347c1c69952894caddbd3" exitCode=0 Nov 03 17:17:37.702154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.702138 4885 manager.go:324] Recovery completed Nov 03 17:17:37.716405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.716384 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.719781 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.719736 4885 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:17:37.721287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.721271 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.721388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.721305 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.721388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.721320 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.725096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725052 4885 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:17:37.725096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725094 4885 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:17:37.725225 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725116 4885 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:17:37.725225 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725125 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.725305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725288 4885 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 17:17:37.725369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725302 4885 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 17:17:37.725369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725333 4885 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 17:17:37.725369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725342 4885 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 17:17:37.725369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725349 4885 policy_none.go:50] "Start" Nov 03 17:17:37.725369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725366 4885 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:17:37.725559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.725379 4885 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:17:37.726467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.726448 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.726542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.726485 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.726542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.726501 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.727043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.727029 4885 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 17:17:37.727108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.727051 4885 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 17:17:37.727108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.727059 4885 policy_none.go:44] "Start" Nov 03 17:17:37.732178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.732152 4885 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:17:37.736441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736426 4885 manager.go:341] "Starting Device Plugin manager" Nov 03 17:17:37.736527 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.736449 4885 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:17:37.736527 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736457 4885 server.go:87] "Starting device plugin registration server" Nov 03 17:17:37.736776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736761 4885 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:17:37.736850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736777 4885 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:17:37.736850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736842 4885 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:17:37.736996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.736938 4885 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:17:37.737104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.737003 4885 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:17:37.737104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.737011 4885 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:17:37.737176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.737082 4885 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:17:37.736989084Z","UUID":"12755303-bc09-4ab6-b9b6-9300521e703e","Handler":null,"Name":"","Endpoint":""} Nov 03 17:17:37.738341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.738322 4885 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:17:37.738436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.738355 4885 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:17:37.738654 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.738636 4885 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:17:37.738739 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.738675 4885 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" Nov 03 17:17:37.756050 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.755604 4885 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal"] Nov 03 17:17:37.756193 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.756156 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.757017 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.757001 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.757140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.757034 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.757140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.757049 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.757367 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.757354 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.757498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.757485 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.758108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758093 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.758183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758095 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.758183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758119 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.758183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758128 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.758183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758140 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.758183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758153 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.758350 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.758287 4885 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.758442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758361 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"31fb60344eae41808f24a7b8d8670170","Type":"ContainerStarted","Data":"7f8bce6e88cdc55735248921950b2bc6b830aa471edb165cdc8d1c4ef4942ea8"} Nov 03 17:17:37.758442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758394 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"31fb60344eae41808f24a7b8d8670170","Type":"ContainerStarted","Data":"9f83a75fff4e7615295d7c5cbe5f2d9e463e6b73ceba328ab12855ad5f0e2794"} Nov 03 17:17:37.758442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758420 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"fd4690c918573c583e31619b50a8a4b6771d4b046c180e20eac576f66a132fe1"} Nov 03 17:17:37.758442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758432 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerDied","Data":"3a7994358f7fee92a17f84628cf3a90399c28ac31a7347c1c69952894caddbd3"} Nov 03 17:17:37.758587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758445 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" event={"ID":"bf9034c461ae30eb2c88e2ba1d28d459","Type":"ContainerStarted","Data":"1ccced3563822ffd9cb67d917daecbd9613b1c8497bb84d3ec33daeaa3c11feb"} Nov 03 17:17:37.758587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.758461 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.759128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.759113 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.759191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.759138 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.759191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.759149 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.759275 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.759266 4885 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-76.us-west-2.compute.internal\" not found" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.825125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.825092 4885 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:17:37.825258 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.825186 4885 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 17:17:37.825589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.825560 4885 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:17:37.825589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.825570 4885 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:17:37.837762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.837743 4885 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:17:37.838736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.838719 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:17:37.838736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.838752 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:17:37.838916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.838766 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:17:37.838916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.838814 4885 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.919107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.919081 4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.926278 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.926253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.926343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.926298 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.926343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.926327 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.927638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.927618 4885 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:17:37.982811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.982747 4885 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.982918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.982820 4885 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.983395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.983375 4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:17:37.983467 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:37.983433 4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.983467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.983447 4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:37.983467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.983452 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeNotReady" Nov 03 17:17:37.983583 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:37.983481 4885 setters.go:546] "Node became not ready" node="ip-10-0-129-76.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T17:17:37Z","lastTransitionTime":"2026-11-03T17:17:37Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 17:17:38.027221 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027195 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.027285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027225 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.027285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027242 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.027357 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027298 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.027357 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027292 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf9034c461ae30eb2c88e2ba1d28d459-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"bf9034c461ae30eb2c88e2ba1d28d459\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.027424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.027361 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/31fb60344eae41808f24a7b8d8670170-config\") pod \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" (UID: \"31fb60344eae41808f24a7b8d8670170\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.045137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.045111 4885 nodeinfomanager.go:471] existing CSINode "ip-10-0-129-76.us-west-2.compute.internal" is owned by different node (oldNodeID="6662698a-2844-4f02-b309-73cb54fefec9", newNodeID=""), cleaning up... Nov 03 17:17:38.082542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.082519 4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:17:38.082648 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:38.082560 4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.605595 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.605563 4885 apiserver.go:52] "Watching apiserver" Nov 03 17:17:38.701147 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.701119 4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.701285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.701196 4885 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.763140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.763107 4885 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:17:38.763594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.763574 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-xlbvs","openshift-ingress-canary/ingress-canary-6rgjt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal","openshift-console/console-fbf6c6fc9-tqhhc","openshift-multus/multus-tvt2q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d","openshift-cluster-node-tuning-operator/tuned-lz759","openshift-dns/node-resolver-xt562","openshift-insights/insights-runtime-extractor-89c8l","openshift-image-registry/node-ca-dw5h9","openshift-monitoring/node-exporter-8mxk9","kube-system/global-pull-secret-syncer-z526b","openshift-multus/multus-additional-cni-plugins-xw9w5","openshift-multus/network-metrics-daemon-l54gp","kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal","openshift-dns/dns-default-jmf7n","openshift-network-diagnostics/network-check-target-j65bb","openshift-network-operator/iptables-alerter-wptbt","openshift-ovn-kubernetes/ovnkube-node-bg64f"] Nov 03 17:17:38.764547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.764506 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:38.777469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.777448 4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:17:38.777577 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:38.777508 4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.778095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.778061 4885 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:17:38.778175 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:38.778123 4885 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-76.us-west-2.compute.internal" Nov 03 17:17:38.820418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.820394 4885 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:17:38.823793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.823776 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.823896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.823838 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:17:38.824163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824141 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:17:38.824212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824188 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:17:38.824212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824198 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwgrr\"" Nov 03 17:17:38.824700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824682 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.824762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824709 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fmcvz\"" Nov 03 17:17:38.824762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824726 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:17:38.824762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824730 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.824762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824759 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.824951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824764 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:17:38.824951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824867 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:17:38.824951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824875 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h6tl4\"" Nov 03 17:17:38.824951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824937 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whvtr\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824957 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824962 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824970 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x9v6p\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.824992 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825003 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825004 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825044 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825084 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825119 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825146 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:17:38.825158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825165 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825170 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czdwp\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825174 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-d8tp8\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825242 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ptg9\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825231 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h78gf\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825416 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825460 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.825513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825501 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.825725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825715 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mr6xr\"" Nov 03 17:17:38.825825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.825812 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:17:38.832048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832033 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.832120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832059 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:38.832120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832092 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.832120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832107 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.832305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832161 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.832305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832198 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgfm\" (UniqueName: \"kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.832305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832229 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.832482 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832274 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.832628 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832609 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.832983 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833088 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833253 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833291 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833326 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833365 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvsv\" (UniqueName: \"kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833402 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvczl\" (UniqueName: \"kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833429 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7b5m\" (UniqueName: \"kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833473 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833523 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833558 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833581 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833612 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833642 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqhl\" (UniqueName: \"kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833671 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5n4\" (UniqueName: \"kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833706 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp9n\" (UniqueName: \"kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833733 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833755 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833786 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833823 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833851 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833889 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833912 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833939 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.833967 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834001 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834028 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834049 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834098 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834126 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834153 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834178 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834214 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834240 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834267 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834296 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834322 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834358 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834383 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834415 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834444 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834474 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834504 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834527 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834555 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834585 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834614 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834639 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834668 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834701 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2bl\" (UniqueName: \"kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834729 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834760 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834787 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834819 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57g6\" (UniqueName: \"kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834849 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834880 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834909 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834935 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834965 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.834994 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835026 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835053 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835101 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835132 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835161 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835222 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835258 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835290 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczxl\" (UniqueName: \"kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835323 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835351 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835376 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835410 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835439 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835468 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.835538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835496 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.835697 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6zw\" (UniqueName: \"kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836164 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836282 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836305 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35b8f38-9b01-4c09-85d8-560e93cd71da-tmp-dir\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836333 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836368 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836367 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-original-pull-secret\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836400 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836486 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836550 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6eec4786-068b-439e-90cd-328eea55a1af-data-volume\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836556 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836600 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cni-binary-copy\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836614 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54863910-20c1-44fe-b7f7-f2a39321a9eb-agent-certs\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836788 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-iptables-alerter-script\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836866 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836932 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-tuned\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836934 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836885 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-textfile\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.836962 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/266c9430-4b51-4065-a2fa-00f638011a0f-metrics-certs\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837078 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837128 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-binary-copy\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837223 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837225 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-daemon-config\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837258 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837293 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837338 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837742 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837815 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837914 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837915 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/37c806b4-ceab-4062-aa31-e0776a71ed05-serviceca\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837927 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d6141b3-1493-4bd1-b637-5d4290f46930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.837969 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838017 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838051 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838118 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838140 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ng6\" (UniqueName: \"kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838191 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838236 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/936ced82-e99e-4fee-8690-70ea861d6890-tmp-dir\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838291 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838327 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838372 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.838589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.838407 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca30f271-2ab1-468f-8413-64c76f5eb172-tmp\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.877486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.877459 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.877589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.877497 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.877589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.877497 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.877589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.877545 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9lkll\"" Nov 03 17:17:38.881078 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881051 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 17:17:38.881169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881135 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:17:38.881169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881144 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:17:38.881169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881153 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 17:17:38.881536 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881521 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.881649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881634 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.881962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881947 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:17:38.882001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.881985 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:17:38.882046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882037 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 17:17:38.882122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882088 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 17:17:38.882122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882102 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jrssc\"" Nov 03 17:17:38.882122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882109 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:17:38.882122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882059 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c6dxq\"" Nov 03 17:17:38.882122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882117 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4vh8j\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882145 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882153 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882159 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882164 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qk7gm\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882177 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gnkq2\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882184 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882290 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9495t\"" Nov 03 17:17:38.882303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882286 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 17:17:38.882513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882327 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.882513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882345 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.882673 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882659 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 17:17:38.882766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882754 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 17:17:38.882808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882799 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:17:38.882838 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882799 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-cert\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:38.882838 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.882828 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:17:38.883057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.883042 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 17:17:38.883153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.883132 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 17:17:38.883153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.883151 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:17:38.884315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.884296 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6eec4786-068b-439e-90cd-328eea55a1af-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886582 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886633 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886642 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6eec4786-068b-439e-90cd-328eea55a1af-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886662 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/936ced82-e99e-4fee-8690-70ea861d6890-config-volume\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886679 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-config\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886669 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-metrics-client-ca\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886692 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.886729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886695 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-tls\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.887027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886757 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovn-node-metrics-cert\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.887027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886792 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.887027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.886797 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54863910-20c1-44fe-b7f7-f2a39321a9eb-konnectivity-ca\") pod \"konnectivity-agent-xlbvs\" (UID: \"54863910-20c1-44fe-b7f7-f2a39321a9eb\") " pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:38.887978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.887961 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/936ced82-e99e-4fee-8690-70ea861d6890-metrics-tls\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.888045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.887979 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.888134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.888118 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.889156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.889143 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-env-overrides\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.893358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.893341 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b6742db-d55d-4aac-b612-dafe9c875cc5-ovnkube-script-lib\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.928282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.928253 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 17:17:38.928406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.928388 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 17:17:38.928698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.928675 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgfm\" (UniqueName: \"kubernetes.io/projected/ca30f271-2ab1-468f-8413-64c76f5eb172-kube-api-access-vqgfm\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.928889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.928869 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvtx\" (UniqueName: \"kubernetes.io/projected/37c806b4-ceab-4062-aa31-e0776a71ed05-kube-api-access-zlvtx\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929200 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5n4\" (UniqueName: \"kubernetes.io/projected/6b6742db-d55d-4aac-b612-dafe9c875cc5-kube-api-access-md5n4\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929272 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqhl\" (UniqueName: \"kubernetes.io/projected/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-kube-api-access-tbqhl\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929274 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvsv\" (UniqueName: \"kubernetes.io/projected/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-kube-api-access-2hvsv\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929440 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7b5m\" (UniqueName: \"kubernetes.io/projected/266c9430-4b51-4065-a2fa-00f638011a0f-kube-api-access-t7b5m\") pod \"network-metrics-daemon-l54gp\" (UID: \"266c9430-4b51-4065-a2fa-00f638011a0f\") " pod="openshift-multus/network-metrics-daemon-l54gp" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929501 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2bl\" (UniqueName: \"kubernetes.io/projected/2d6141b3-1493-4bd1-b637-5d4290f46930-kube-api-access-hn2bl\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.929548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929534 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6zw\" (UniqueName: \"kubernetes.io/projected/e35b8f38-9b01-4c09-85d8-560e93cd71da-kube-api-access-2x6zw\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.929906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929534 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp9n\" (UniqueName: \"kubernetes.io/projected/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kube-api-access-bjp9n\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.929906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929695 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvczl\" (UniqueName: \"kubernetes.io/projected/6e87cb76-bc36-418c-b7fd-393ccbe2d8db-kube-api-access-nvczl\") pod \"ingress-canary-6rgjt\" (UID: \"6e87cb76-bc36-418c-b7fd-393ccbe2d8db\") " pod="openshift-ingress-canary/ingress-canary-6rgjt" Nov 03 17:17:38.929906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929742 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57g6\" (UniqueName: \"kubernetes.io/projected/6eec4786-068b-439e-90cd-328eea55a1af-kube-api-access-v57g6\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.929906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929757 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ng6\" (UniqueName: \"kubernetes.io/projected/936ced82-e99e-4fee-8690-70ea861d6890-kube-api-access-d6ng6\") pod \"dns-default-jmf7n\" (UID: \"936ced82-e99e-4fee-8690-70ea861d6890\") " pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:38.929906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.929806 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczxl\" (UniqueName: \"kubernetes.io/projected/b7b90722-afde-44ca-88ac-a73e3b3c61aa-kube-api-access-sczxl\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.930184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.930147 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.936712 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.936694 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.938498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938478 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"console-fbf6c6fc9-tqhhc\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:38.938645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938632 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.938681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938652 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938668 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.938735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938695 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.938735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938718 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.938735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938723 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-systemd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938751 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-root\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938767 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysconfig\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938775 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938756 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938801 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-modprobe-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938811 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-node-exporter-wtmp\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.938818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938809 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938838 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-run-ovn\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938838 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938858 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-system-cni-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938863 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938901 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938911 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938937 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938951 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-socket-dir-parent\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938960 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.938996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938985 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.938997 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-var-lib-kubelet\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939010 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939013 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-slash\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939032 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939051 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-kubernetes\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939033 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-bin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939058 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939122 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939142 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939161 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-d\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939166 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939171 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939195 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939198 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-socket-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939204 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-log-socket\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939203 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-registration-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939222 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-cni-multus\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939221 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939255 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939269 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939276 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-kubelet\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939293 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939297 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-var-lib-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.939300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939307 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-sysctl-conf\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939318 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939325 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-kubelet-config\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939333 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939336 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-multus-certs\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939349 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939364 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939378 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939375 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-etc-openvswitch\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939409 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-k8s-cni-cncf-io\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939415 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-lib-modules\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939414 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb-dbus\") pod \"global-pull-secret-syncer-z526b\" (UID: \"9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb\") " pod="kube-system/global-pull-secret-syncer-z526b" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939435 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939452 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939469 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939477 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-host\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939491 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939508 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-netns\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939514 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939531 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-var-lib-kubelet\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939536 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939551 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939568 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939513 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939585 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-multus-conf-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939609 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-etc-kubernetes\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939629 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939628 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-etc-systemd\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939641 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-node-log\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939648 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939663 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d6141b3-1493-4bd1-b637-5d4290f46930-cnibin\") pod \"multus-additional-cni-plugins-xw9w5\" (UID: \"2d6141b3-1493-4bd1-b637-5d4290f46930\") " pod="openshift-multus/multus-additional-cni-plugins-xw9w5" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939663 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-cnibin\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939665 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939683 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-run-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939697 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939716 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939734 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939719 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-sys\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939766 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939747 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-systemd-units\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939740 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-bin\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939793 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939806 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939820 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939834 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939837 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b6742db-d55d-4aac-b612-dafe9c875cc5-host-cni-netd\") pod \"ovnkube-node-bg64f\" (UID: \"6b6742db-d55d-4aac-b612-dafe9c875cc5\") " pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939856 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939869 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-sys-fs\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939874 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939878 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/37c806b4-ceab-4062-aa31-e0776a71ed05-host\") pod \"node-ca-dw5h9\" (UID: \"37c806b4-ceab-4062-aa31-e0776a71ed05\") " pod="openshift-image-registry/node-ca-dw5h9" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939893 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb8b9a87-1619-4d2c-8989-3bba620ce5b5-host-slash\") pod \"iptables-alerter-wptbt\" (UID: \"bb8b9a87-1619-4d2c-8989-3bba620ce5b5\") " pod="openshift-network-operator/iptables-alerter-wptbt" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939893 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939926 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6eec4786-068b-439e-90cd-328eea55a1af-crio-socket\") pod \"insights-runtime-extractor-89c8l\" (UID: \"6eec4786-068b-439e-90cd-328eea55a1af\") " pod="openshift-insights/insights-runtime-extractor-89c8l" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939928 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-hostroot\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939946 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939971 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.939996 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940015 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7b90722-afde-44ca-88ac-a73e3b3c61aa-sys\") pod \"node-exporter-8mxk9\" (UID: \"b7b90722-afde-44ca-88ac-a73e3b3c61aa\") " pod="openshift-monitoring/node-exporter-8mxk9" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940022 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ca30f271-2ab1-468f-8413-64c76f5eb172-run\") pod \"tuned-lz759\" (UID: \"ca30f271-2ab1-468f-8413-64c76f5eb172\") " pod="openshift-cluster-node-tuning-operator/tuned-lz759" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940019 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940042 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.940026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940054 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-host-run-netns\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940059 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940105 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-device-dir\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940105 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f39eb672-c5af-4f63-b7ef-b34fa3674c6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgd7d\" (UID: \"f39eb672-c5af-4f63-b7ef-b34fa3674c6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgd7d" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940127 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940123 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e35b8f38-9b01-4c09-85d8-560e93cd71da-hosts-file\") pod \"node-resolver-xt562\" (UID: \"e35b8f38-9b01-4c09-85d8-560e93cd71da\") " pod="openshift-dns/node-resolver-xt562" Nov 03 17:17:38.941734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:38.940194 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b53025bb-d955-4dd9-a75e-74ad4bdaf61b-system-cni-dir\") pod \"multus-tvt2q\" (UID: \"b53025bb-d955-4dd9-a75e-74ad4bdaf61b\") " pod="openshift-multus/multus-tvt2q" Nov 03 17:17:39.000371 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.000333 4885 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:17:39.002681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.002663 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mx59\" (UniqueName: \"kubernetes.io/projected/83f65e07-2574-44d8-9fe0-de9b7f689bd4-kube-api-access-4mx59\") pod \"network-check-target-j65bb\" (UID: \"83f65e07-2574-44d8-9fe0-de9b7f689bd4\") " pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:39.065086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.065033 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:39.415909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.415882 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:39.427092 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:39.427046 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f65e07_2574_44d8_9fe0_de9b7f689bd4.slice/crio-b5cf16a2e71d7edda002643fa5355143a22d23f077e9f8ba7203d80751033be7 WatchSource:0}: Error finding container b5cf16a2e71d7edda002643fa5355143a22d23f077e9f8ba7203d80751033be7: Status 404 returned error can't find the container with id b5cf16a2e71d7edda002643fa5355143a22d23f077e9f8ba7203d80751033be7 Nov 03 17:17:39.427898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.427875 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j65bb"] Nov 03 17:17:39.429563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.429546 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:17:39.707558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:39.707470 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j65bb" event={"ID":"83f65e07-2574-44d8-9fe0-de9b7f689bd4","Type":"ContainerStarted","Data":"b5cf16a2e71d7edda002643fa5355143a22d23f077e9f8ba7203d80751033be7"} Nov 03 17:17:41.121384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:41.121355 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:17:41.121849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:41.121544 4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:17:42.013282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:42.013253 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jmf7n" Nov 03 17:17:42.036988 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:42.036949 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:42.041004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:42.040975 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:42.719704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:42.719666 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j65bb" event={"ID":"83f65e07-2574-44d8-9fe0-de9b7f689bd4","Type":"ContainerStarted","Data":"f88e77a733748d07708dc8cf38b248b8e2e84bb9084cbbe12a0247fe790c6c69"} Nov 03 17:17:43.722849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:43.722820 4885 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:17:45.668949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:45.668919 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:17:46.359443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:46.359406 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:17:48.114788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:48.114756 4885 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-76.us-west-2.compute.internal" event="NodeReady" Nov 03 17:17:48.545645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:48.545616 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xlbvs" Nov 03 17:17:48.604543 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:17:48.604501 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:17:54.004684 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.004650 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-85fb8b6678-k4crd"] Nov 03 17:17:54.010017 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.010000 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.115927 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.115898 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fb8b6678-k4crd"] Nov 03 17:17:54.147810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.147783 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-service-ca\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.147810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.147819 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-trusted-ca-bundle\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148006 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.147844 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-oauth-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148006 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.147974 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3516-ae41-49c0-94bf-82a1ff556e34-tmp\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.148008 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-oauth-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.148033 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xgxt\" (UniqueName: \"kubernetes.io/projected/ebbf3516-ae41-49c0-94bf-82a1ff556e34-kube-api-access-9xgxt\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.148133 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.148168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.148149 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249246 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xgxt\" (UniqueName: \"kubernetes.io/projected/ebbf3516-ae41-49c0-94bf-82a1ff556e34-kube-api-access-9xgxt\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249290 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249306 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249325 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-service-ca\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249468 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-trusted-ca-bundle\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249515 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-oauth-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249555 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3516-ae41-49c0-94bf-82a1ff556e34-tmp\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249591 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-oauth-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.249989 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.249948 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebbf3516-ae41-49c0-94bf-82a1ff556e34-tmp\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.250144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.250022 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.250144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.250108 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-service-ca\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.250275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.250255 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-oauth-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.251080 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.251043 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ebbf3516-ae41-49c0-94bf-82a1ff556e34-trusted-ca-bundle\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.251898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.251873 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-serving-cert\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.252209 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.252191 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebbf3516-ae41-49c0-94bf-82a1ff556e34-console-oauth-config\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.312420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.312353 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xgxt\" (UniqueName: \"kubernetes.io/projected/ebbf3516-ae41-49c0-94bf-82a1ff556e34-kube-api-access-9xgxt\") pod \"console-85fb8b6678-k4crd\" (UID: \"ebbf3516-ae41-49c0-94bf-82a1ff556e34\") " pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.321194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.321173 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:17:54.688748 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:17:54.688716 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbf3516_ae41_49c0_94bf_82a1ff556e34.slice/crio-f6d3d2fe4b89c5aa11a40f49dec3db13a77907e0d3f9680a3d10d579a71a80a2 WatchSource:0}: Error finding container f6d3d2fe4b89c5aa11a40f49dec3db13a77907e0d3f9680a3d10d579a71a80a2: Status 404 returned error can't find the container with id f6d3d2fe4b89c5aa11a40f49dec3db13a77907e0d3f9680a3d10d579a71a80a2 Nov 03 17:17:54.689912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.689891 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85fb8b6678-k4crd"] Nov 03 17:17:54.758739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.758710 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fb8b6678-k4crd" event={"ID":"ebbf3516-ae41-49c0-94bf-82a1ff556e34","Type":"ContainerStarted","Data":"6eb167b75f79cfbb420e2785e08984715f162d30c5adc0cab46f06f6a87181c6"} Nov 03 17:17:54.758871 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.758763 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85fb8b6678-k4crd" event={"ID":"ebbf3516-ae41-49c0-94bf-82a1ff556e34","Type":"ContainerStarted","Data":"f6d3d2fe4b89c5aa11a40f49dec3db13a77907e0d3f9680a3d10d579a71a80a2"} Nov 03 17:17:54.881412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:54.881357 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-85fb8b6678-k4crd" podStartSLOduration=1.881344859 podStartE2EDuration="1.881344859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:54.880712216 +0000 UTC m=+17.335401242" watchObservedRunningTime="2026-11-03 17:17:54.881344859 +0000 UTC m=+17.336033881" Nov 03 17:17:56.364103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:17:56.364054 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:18:01.797130 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.797097 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:18:01.839536 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.839502 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.868422 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.868391 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:18:01.898152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898123 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898155 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-config-out\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898179 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898232 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898263 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898344 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898360 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898395 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898440 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq68n\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-kube-api-access-nq68n\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898479 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898494 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898541 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.898588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.898559 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-web-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996667 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996667 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996704 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996732 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996739 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996752 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996767 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996768 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 17:18:01.996814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996789 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rppdf\"" Nov 03 17:18:01.997266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.996820 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 17:18:01.999304 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999286 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999313 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-web-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999335 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999356 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-config-out\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999389 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999513 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999551 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999656 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999600 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999656 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999630 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999779 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999694 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999779 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999720 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq68n\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-kube-api-access-nq68n\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999779 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999747 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999779 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999771 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:01.999972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:01.999784 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.000793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.000413 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.002636 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.002605 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2e737e2-92b0-41e7-bdd5-826b84824632-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003060 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003173 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003291 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003303 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003348 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.003811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.003466 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-web-config\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.004332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.004308 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.004481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.004465 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2e737e2-92b0-41e7-bdd5-826b84824632-config-out\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.004618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.004598 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2e737e2-92b0-41e7-bdd5-826b84824632-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.063926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.063889 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq68n\" (UniqueName: \"kubernetes.io/projected/f2e737e2-92b0-41e7-bdd5-826b84824632-kube-api-access-nq68n\") pod \"alertmanager-main-0\" (UID: \"f2e737e2-92b0-41e7-bdd5-826b84824632\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.251129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.251022 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:18:02.749706 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:18:02.749670 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e737e2_92b0_41e7_bdd5_826b84824632.slice/crio-493bcaa1c9720ea54abbe8705a4bf9536b4edd52556dd2c1437b3ac1912f16e9 WatchSource:0}: Error finding container 493bcaa1c9720ea54abbe8705a4bf9536b4edd52556dd2c1437b3ac1912f16e9: Status 404 returned error can't find the container with id 493bcaa1c9720ea54abbe8705a4bf9536b4edd52556dd2c1437b3ac1912f16e9 Nov 03 17:18:02.749829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.749813 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:18:02.786954 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.786921 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"493bcaa1c9720ea54abbe8705a4bf9536b4edd52556dd2c1437b3ac1912f16e9"} Nov 03 17:18:02.896188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:02.896161 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bg64f" Nov 03 17:18:04.322280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.322243 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:18:04.322742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.322316 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:18:04.327048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.327027 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:18:04.794295 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.794250 4885 generic.go:356] "Generic (PLEG): container finished" podID="f2e737e2-92b0-41e7-bdd5-826b84824632" containerID="9e97179f5e6244f4bb41756e5e4c9d6151cc08689525dbe734e27d30cf393089" exitCode=0 Nov 03 17:18:04.794487 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.794318 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerDied","Data":"9e97179f5e6244f4bb41756e5e4c9d6151cc08689525dbe734e27d30cf393089"} Nov 03 17:18:04.798384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:04.798367 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85fb8b6678-k4crd" Nov 03 17:18:05.100814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:05.100735 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:18:06.805713 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:06.805631 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"609d7951a8e817e751fc9328fe050d34a3bd507b9bc35ef969843b8e0df83f48"} Nov 03 17:18:06.805713 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:06.805687 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"866b5be04a457e2275b0c2c034db648b604e6241e27134702c55c528ee7ce0b3"} Nov 03 17:18:07.810864 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:07.810828 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"df127cbe33ef709b214da40657c15b5cdbfc6cc7475a13c6f0e45d86328fd2ef"} Nov 03 17:18:07.811254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:07.810889 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"d88fe3208d95be8b1aac95592f05e231d65f0ab462020f420276e37cbb5c7d36"} Nov 03 17:18:07.811254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:07.810912 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"fc8ccbb8c642d7b46cb3639512bf2053d1a4298092a59903b837af976571060c"} Nov 03 17:18:08.817157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:08.817122 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2e737e2-92b0-41e7-bdd5-826b84824632","Type":"ContainerStarted","Data":"7cca905587db8c5dac2514721da8e14dae01a4a0ec999d4265a872b52f11ab8d"} Nov 03 17:18:09.092580 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:09.092470 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.196392646 podStartE2EDuration="8.092452297s" totalImagesPullingTime="4.88174467s" totalInitContainerRuntime="14.314981ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:18:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:18:09.091759643 +0000 UTC m=+31.546448667" watchObservedRunningTime="2026-11-03 17:18:09.092452297 +0000 UTC m=+31.547141322" Nov 03 17:18:15.671309 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:15.671279 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j65bb" Nov 03 17:18:18.604058 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:18:18.604020 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:18:30.120755 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.120696 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fbf6c6fc9-tqhhc" podUID="0c54a443-2e3f-4b1c-82b5-340188cae67a" containerName="console" containerID="cri-o://74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49" gracePeriod=15 Nov 03 17:18:30.371414 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.371365 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:18:30.418384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418357 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418402 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418438 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418459 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418478 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418500 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418538 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418582 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") pod \"0c54a443-2e3f-4b1c-82b5-340188cae67a\" (UID: \"0c54a443-2e3f-4b1c-82b5-340188cae67a\") " Nov 03 17:18:30.418779 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418666 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:18:30.418837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418813 4885 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c54a443-2e3f-4b1c-82b5-340188cae67a-tmp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.418893 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418861 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:30.418992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418967 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:30.418992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418975 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:30.419100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.418987 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:18:30.421046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.421021 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:18:30.421175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.421103 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "kube-api-access-br2d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:18:30.421175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.421110 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config" pod "0c54a443-2e3f-4b1c-82b5-340188cae67a" (UID: "0c54a443-2e3f-4b1c-82b5-340188cae67a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:18:30.520031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520000 4885 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-oauth-config\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520028 4885 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-trusted-ca-bundle\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520038 4885 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-config\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520047 4885 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-oauth-serving-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520056 4885 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0c54a443-2e3f-4b1c-82b5-340188cae67a-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520084 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br2d5\" (UniqueName: \"kubernetes.io/projected/0c54a443-2e3f-4b1c-82b5-340188cae67a-kube-api-access-br2d5\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.520273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.520096 4885 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c54a443-2e3f-4b1c-82b5-340188cae67a-console-serving-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:18:30.887828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.887797 4885 generic.go:356] "Generic (PLEG): container finished" podID="0c54a443-2e3f-4b1c-82b5-340188cae67a" containerID="74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49" exitCode=0 Nov 03 17:18:30.887987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.887844 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fbf6c6fc9-tqhhc" Nov 03 17:18:30.887987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.887874 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbf6c6fc9-tqhhc" event={"ID":"0c54a443-2e3f-4b1c-82b5-340188cae67a","Type":"ContainerDied","Data":"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49"} Nov 03 17:18:30.887987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.887924 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fbf6c6fc9-tqhhc" event={"ID":"0c54a443-2e3f-4b1c-82b5-340188cae67a","Type":"ContainerDied","Data":"b5f82a399b329e3eb8ab09f0684b063453c34f27b26426dc76aa1d5345f1af18"} Nov 03 17:18:30.887987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.887945 4885 scope.go:122] "RemoveContainer" containerID="74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49" Nov 03 17:18:30.896118 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.896102 4885 scope.go:122] "RemoveContainer" containerID="74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49" Nov 03 17:18:30.896405 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:18:30.896382 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49\": container with ID starting with 74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49 not found: ID does not exist" containerID="74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49" Nov 03 17:18:30.896495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:30.896409 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49"} err="failed to get container status \"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49\": rpc error: code = NotFound desc = could not find container \"74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49\": container with ID starting with 74e83466113b08a8358fcafefbce985cf1bdae3e5b32d0c0d3a392a7b0959b49 not found: ID does not exist" Nov 03 17:18:31.056911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:31.056876 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:18:31.108038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:31.108006 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fbf6c6fc9-tqhhc"] Nov 03 17:18:31.659593 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:18:31.659526 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c54a443-2e3f-4b1c-82b5-340188cae67a" path="/var/lib/kubelet/pods/0c54a443-2e3f-4b1c-82b5-340188cae67a/volumes" Nov 03 17:18:48.604735 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:18:48.604693 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:19:18.604379 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:19:18.604337 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:19:48.604505 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:19:48.604465 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:20:18.604289 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:20:18.604249 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:20:33.638655 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.638619 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425"] Nov 03 17:20:33.639036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.638869 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0c54a443-2e3f-4b1c-82b5-340188cae67a" containerName="console" Nov 03 17:20:33.639036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.638879 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c54a443-2e3f-4b1c-82b5-340188cae67a" containerName="console" Nov 03 17:20:33.639036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.638936 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c54a443-2e3f-4b1c-82b5-340188cae67a" containerName="console" Nov 03 17:20:33.640845 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.640831 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.667085 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.667039 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhsn\" (UniqueName: \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.667204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.667133 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.667243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.667205 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.694827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.694805 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 17:20:33.746200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.746177 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 17:20:33.746323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.746217 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-xwb2x\"" Nov 03 17:20:33.746892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.746871 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425"] Nov 03 17:20:33.767607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.767588 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.767667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.767621 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.767667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.767655 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhsn\" (UniqueName: \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.767969 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.767952 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.768002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.767976 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.830893 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.830862 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhsn\" (UniqueName: \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.861817 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.861794 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:20:33.864011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.863998 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.930103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.930020 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 17:20:33.932897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.932856 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 17:20:33.934557 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.934531 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:20:33.935155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.935137 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 17:20:33.935844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.935829 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 17:20:33.935995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.935978 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-685cm\"" Nov 03 17:20:33.936095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.936082 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 17:20:33.953371 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.953353 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:33.968716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.968689 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.968853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.968740 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgmp\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.968853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.968787 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.968975 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.968956 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.969036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.969001 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.969122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.969103 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:33.969188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:33.969156 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.030748 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.030723 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 17:20:34.070243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070217 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070252 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070274 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgmp\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070454 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070478 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070498 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070513 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.070846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.070824 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.072693 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.072674 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.072940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.072910 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.073026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.072992 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.073095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.073057 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.133121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.133096 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgmp\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.133121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.133120 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hgvzt\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.175200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.175177 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:34.321810 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:20:34.321776 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f726c1_ea39_4ad0_8fe9_7188cf0f3338.slice/crio-08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c WatchSource:0}: Error finding container 08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c: Status 404 returned error can't find the container with id 08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c Nov 03 17:20:34.322362 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.322343 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425"] Nov 03 17:20:34.546056 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:20:34.546030 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647b58ea_93f7_4b5e_bd2c_872ccde7695c.slice/crio-1c48bc9074837439f9aa3a0973db8fba79e45178fdbc3d1fefd70a2444c2a98f WatchSource:0}: Error finding container 1c48bc9074837439f9aa3a0973db8fba79e45178fdbc3d1fefd70a2444c2a98f: Status 404 returned error can't find the container with id 1c48bc9074837439f9aa3a0973db8fba79e45178fdbc3d1fefd70a2444c2a98f Nov 03 17:20:34.546333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:34.546313 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:20:35.272951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:35.272912 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" event={"ID":"17f726c1-ea39-4ad0-8fe9-7188cf0f3338","Type":"ContainerStarted","Data":"08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c"} Nov 03 17:20:35.273981 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:35.273957 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" event={"ID":"647b58ea-93f7-4b5e-bd2c-872ccde7695c","Type":"ContainerStarted","Data":"1c48bc9074837439f9aa3a0973db8fba79e45178fdbc3d1fefd70a2444c2a98f"} Nov 03 17:20:38.663166 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:38.663128 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:20:38.663525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:38.663202 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:20:39.294593 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:39.294553 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" event={"ID":"647b58ea-93f7-4b5e-bd2c-872ccde7695c","Type":"ContainerStarted","Data":"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4"} Nov 03 17:20:39.294817 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:39.294795 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:39.296836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:39.296547 4885 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hgvzt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:20:39.296836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:39.296610 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:20:40.299610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:40.299582 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:20:40.434674 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:40.434613 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" podStartSLOduration=3.320204722 podStartE2EDuration="7.43459469s" totalImagesPullingTime="4.114389968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:20:39.510478664 +0000 UTC m=+181.965167690" watchObservedRunningTime="2026-11-03 17:20:40.43459469 +0000 UTC m=+182.889283716" Nov 03 17:20:41.303606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:41.303571 4885 generic.go:356] "Generic (PLEG): container finished" podID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerID="20e8fa65d6d72056f5fc739c8c42e41a87910af097a34f328cddc13508447618" exitCode=0 Nov 03 17:20:41.303980 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:41.303662 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" event={"ID":"17f726c1-ea39-4ad0-8fe9-7188cf0f3338","Type":"ContainerDied","Data":"20e8fa65d6d72056f5fc739c8c42e41a87910af097a34f328cddc13508447618"} Nov 03 17:20:45.318819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:45.318784 4885 generic.go:356] "Generic (PLEG): container finished" podID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerID="b904a53f805546e4db47904c21a29ba6623dc528fb317112b2130b7bb3c98ea2" exitCode=0 Nov 03 17:20:45.319274 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:45.318842 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" event={"ID":"17f726c1-ea39-4ad0-8fe9-7188cf0f3338","Type":"ContainerDied","Data":"b904a53f805546e4db47904c21a29ba6623dc528fb317112b2130b7bb3c98ea2"} Nov 03 17:20:48.610053 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:20:48.610006 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:20:52.346401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:52.346358 4885 generic.go:356] "Generic (PLEG): container finished" podID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerID="f9b4cff9b3da5c27283c615ffecd7baa7f8762104c1622124334ec3aaf7d32fa" exitCode=0 Nov 03 17:20:52.346807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:52.346415 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" event={"ID":"17f726c1-ea39-4ad0-8fe9-7188cf0f3338","Type":"ContainerDied","Data":"f9b4cff9b3da5c27283c615ffecd7baa7f8762104c1622124334ec3aaf7d32fa"} Nov 03 17:20:53.465082 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.465044 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:20:53.530555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.530528 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\") pod \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " Nov 03 17:20:53.530702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.530589 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\" (UniqueName: \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\") pod \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " Nov 03 17:20:53.530702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.530615 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\") pod \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\" (UID: \"17f726c1-ea39-4ad0-8fe9-7188cf0f3338\") " Nov 03 17:20:53.531219 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.531184 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle" pod "17f726c1-ea39-4ad0-8fe9-7188cf0f3338" (UID: "17f726c1-ea39-4ad0-8fe9-7188cf0f3338"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:53.532891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.532865 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn" pod "17f726c1-ea39-4ad0-8fe9-7188cf0f3338" (UID: "17f726c1-ea39-4ad0-8fe9-7188cf0f3338"). InnerVolumeSpecName "kube-api-access-pmhsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:53.534849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.534825 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util" pod "17f726c1-ea39-4ad0-8fe9-7188cf0f3338" (UID: "17f726c1-ea39-4ad0-8fe9-7188cf0f3338"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:53.631225 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.631191 4885 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-util\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:20:53.631225 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.631223 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhsn\" (UniqueName: \"kubernetes.io/projected/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-kube-api-access-pmhsn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:20:53.631411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:53.631239 4885 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17f726c1-ea39-4ad0-8fe9-7188cf0f3338-bundle\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:20:54.354806 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:54.354769 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" event={"ID":"17f726c1-ea39-4ad0-8fe9-7188cf0f3338","Type":"ContainerDied","Data":"08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c"} Nov 03 17:20:54.354969 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:54.354826 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a547c4d649e1901f288325a9d157bf714f13b5a3aae0b44ecc29aa900deb9c" Nov 03 17:20:54.354969 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:20:54.354781 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8m425" Nov 03 17:21:18.410101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410009 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd"] Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410295 4885 cpu_manager.go:414] "containerMap: removing container" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="util" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410307 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="util" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410322 4885 cpu_manager.go:414] "containerMap: removing container" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="extract" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410331 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="extract" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410343 4885 cpu_manager.go:414] "containerMap: removing container" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="pull" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410348 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="pull" Nov 03 17:21:18.410477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.410389 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f726c1-ea39-4ad0-8fe9-7188cf0f3338" containerName="extract" Nov 03 17:21:18.412466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.412444 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.468632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.468599 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 17:21:18.468788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.468669 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 17:21:18.468788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.468757 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 17:21:18.512098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512046 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.512253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512115 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.512253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512208 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.512354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512261 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.512354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512347 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sghd\" (UniqueName: \"kubernetes.io/projected/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-kube-api-access-5sghd\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.512441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.512368 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.519458 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.519428 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n7pls\"" Nov 03 17:21:18.519722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.519706 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 17:21:18.519850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.519808 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 17:21:18.521211 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.521194 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd"] Nov 03 17:21:18.604643 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:21:18.604609 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:21:18.613158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613139 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sghd\" (UniqueName: \"kubernetes.io/projected/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-kube-api-access-5sghd\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613168 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613189 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613246 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613300 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613340 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.613563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.613542 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-manifests\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.615638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.615610 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-tmp\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.615756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.615718 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.615921 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.615899 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.616587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.616566 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.685316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.685250 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sghd\" (UniqueName: \"kubernetes.io/projected/cf42bf72-d369-43a8-80ca-4c6bcb1b28e2-kube-api-access-5sghd\") pod \"opendatahub-operator-controller-manager-5cf8664c87-c2nkd\" (UID: \"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:18.725229 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:18.725201 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:19.227502 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:21:19.227457 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf42bf72_d369_43a8_80ca_4c6bcb1b28e2.slice/crio-646ffb9524392aa505093cb586ad6eb2e183a067e5905bc85f24dc9650716757 WatchSource:0}: Error finding container 646ffb9524392aa505093cb586ad6eb2e183a067e5905bc85f24dc9650716757: Status 404 returned error can't find the container with id 646ffb9524392aa505093cb586ad6eb2e183a067e5905bc85f24dc9650716757 Nov 03 17:21:19.227978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:19.227953 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd"] Nov 03 17:21:19.432736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:19.432701 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" event={"ID":"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2","Type":"ContainerStarted","Data":"646ffb9524392aa505093cb586ad6eb2e183a067e5905bc85f24dc9650716757"} Nov 03 17:21:23.449745 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:23.449660 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" event={"ID":"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2","Type":"ContainerStarted","Data":"cc6ed459227938581192cfbbfafa7b402c365d6b869970b642104b8ab91b82ff"} Nov 03 17:21:24.454417 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:24.454379 4885 generic.go:356] "Generic (PLEG): container finished" podID="cf42bf72-d369-43a8-80ca-4c6bcb1b28e2" containerID="cc6ed459227938581192cfbbfafa7b402c365d6b869970b642104b8ab91b82ff" exitCode=0 Nov 03 17:21:24.454814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:24.454436 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" event={"ID":"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2","Type":"ContainerDied","Data":"cc6ed459227938581192cfbbfafa7b402c365d6b869970b642104b8ab91b82ff"} Nov 03 17:21:25.459698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:25.459663 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" event={"ID":"cf42bf72-d369-43a8-80ca-4c6bcb1b28e2","Type":"ContainerStarted","Data":"ba417aa8fb84e79007d165a6d95d20edea96fde72c9f268810cc17415b49c1d1"} Nov 03 17:21:25.460096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:25.459803 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:25.656552 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:25.656505 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" podStartSLOduration=2.390600523 podStartE2EDuration="7.656492443s" totalImagesPullingTime="4.826692494s" totalInitContainerRuntime="439.199426ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:25.647828337 +0000 UTC m=+228.102517362" watchObservedRunningTime="2026-11-03 17:21:25.656492443 +0000 UTC m=+228.111181465" Nov 03 17:21:28.226641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.226612 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:21:28.228694 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.228678 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.253688 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.253663 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:21:28.293406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.293369 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.293539 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.293482 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9ks\" (UniqueName: \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.303908 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.303888 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 17:21:28.303908 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.303906 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 17:21:28.304034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.303937 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f7pq2\"" Nov 03 17:21:28.394734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.394701 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9ks\" (UniqueName: \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.394897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.394755 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.395118 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.395100 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.460930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.460891 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9ks\" (UniqueName: \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\") pod \"kuadrant-operator-controller-manager-cf8469d7b-nhfdb\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.539636 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.539549 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:28.955698 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:21:28.955664 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d233ec7_bec3_41a2_a1e2_aa9c31de4903.slice/crio-b34caa3ef75847da4fad6b7e7377bf0742431d292f4e79b9c26309a93f57dc49 WatchSource:0}: Error finding container b34caa3ef75847da4fad6b7e7377bf0742431d292f4e79b9c26309a93f57dc49: Status 404 returned error can't find the container with id b34caa3ef75847da4fad6b7e7377bf0742431d292f4e79b9c26309a93f57dc49 Nov 03 17:21:28.957783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:28.957761 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:21:29.474925 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:29.474892 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" event={"ID":"6d233ec7-bec3-41a2-a1e2-aa9c31de4903","Type":"ContainerStarted","Data":"b34caa3ef75847da4fad6b7e7377bf0742431d292f4e79b9c26309a93f57dc49"} Nov 03 17:21:31.959496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:31.959463 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc"] Nov 03 17:21:31.968316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:31.968280 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:32.017999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.017969 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nnz\" (UniqueName: \"kubernetes.io/projected/7e51fea8-9d06-422e-9176-70c40477d2ab-kube-api-access-w2nnz\") pod \"dns-operator-controller-manager-8557cbdb84-xbdkc\" (UID: \"7e51fea8-9d06-422e-9176-70c40477d2ab\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:32.089839 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.089804 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 17:21:32.089999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.089903 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xfp4v\"" Nov 03 17:21:32.090929 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.090899 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc"] Nov 03 17:21:32.120971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.120935 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nnz\" (UniqueName: \"kubernetes.io/projected/7e51fea8-9d06-422e-9176-70c40477d2ab-kube-api-access-w2nnz\") pod \"dns-operator-controller-manager-8557cbdb84-xbdkc\" (UID: \"7e51fea8-9d06-422e-9176-70c40477d2ab\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:32.190145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.190110 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nnz\" (UniqueName: \"kubernetes.io/projected/7e51fea8-9d06-422e-9176-70c40477d2ab-kube-api-access-w2nnz\") pod \"dns-operator-controller-manager-8557cbdb84-xbdkc\" (UID: \"7e51fea8-9d06-422e-9176-70c40477d2ab\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:32.282769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.282683 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:32.683970 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:32.683929 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc"] Nov 03 17:21:32.686226 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:21:32.686195 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e51fea8_9d06_422e_9176_70c40477d2ab.slice/crio-20834bae947f8cf5b9575f8bd0e4367e3d9ca0c99ed427fb17fa582ff4f7fd36 WatchSource:0}: Error finding container 20834bae947f8cf5b9575f8bd0e4367e3d9ca0c99ed427fb17fa582ff4f7fd36: Status 404 returned error can't find the container with id 20834bae947f8cf5b9575f8bd0e4367e3d9ca0c99ed427fb17fa582ff4f7fd36 Nov 03 17:21:33.493950 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:33.493917 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" event={"ID":"7e51fea8-9d06-422e-9176-70c40477d2ab","Type":"ContainerStarted","Data":"20834bae947f8cf5b9575f8bd0e4367e3d9ca0c99ed427fb17fa582ff4f7fd36"} Nov 03 17:21:35.505691 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:35.505653 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" event={"ID":"6d233ec7-bec3-41a2-a1e2-aa9c31de4903","Type":"ContainerStarted","Data":"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4"} Nov 03 17:21:35.506194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:35.506172 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:35.733793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:35.733753 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" podStartSLOduration=1.949894045 podStartE2EDuration="7.733738698s" totalImagesPullingTime="5.783844653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:35.733380974 +0000 UTC m=+238.188069998" watchObservedRunningTime="2026-11-03 17:21:35.733738698 +0000 UTC m=+238.188427722" Nov 03 17:21:36.466453 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:36.466423 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cf8664c87-c2nkd" Nov 03 17:21:36.511919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:36.511878 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" event={"ID":"7e51fea8-9d06-422e-9176-70c40477d2ab","Type":"ContainerStarted","Data":"a604b448ed7b3985ef3ce15b487bfb2a785e94d4703b5fd6fc0b52f68da9ab5a"} Nov 03 17:21:36.512338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:36.511966 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:36.848097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:36.847985 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" podStartSLOduration=2.814457004 podStartE2EDuration="5.847970657s" totalImagesPullingTime="3.033513653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:36.846567134 +0000 UTC m=+239.301256159" watchObservedRunningTime="2026-11-03 17:21:36.847970657 +0000 UTC m=+239.302659681" Nov 03 17:21:37.955083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:37.955042 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm"] Nov 03 17:21:37.959590 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:37.959571 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:37.965738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:37.965404 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2cf\" (UniqueName: \"kubernetes.io/projected/b9b78432-2478-417e-9226-2d987a705dc5-kube-api-access-xx2cf\") pod \"authorino-operator-7b7bb4bc77-4lcxm\" (UID: \"b9b78432-2478-417e-9226-2d987a705dc5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:38.018999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.018865 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm"] Nov 03 17:21:38.053721 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.053696 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-597x4\"" Nov 03 17:21:38.065836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.065810 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2cf\" (UniqueName: \"kubernetes.io/projected/b9b78432-2478-417e-9226-2d987a705dc5-kube-api-access-xx2cf\") pod \"authorino-operator-7b7bb4bc77-4lcxm\" (UID: \"b9b78432-2478-417e-9226-2d987a705dc5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:38.134270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.134237 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2cf\" (UniqueName: \"kubernetes.io/projected/b9b78432-2478-417e-9226-2d987a705dc5-kube-api-access-xx2cf\") pod \"authorino-operator-7b7bb4bc77-4lcxm\" (UID: \"b9b78432-2478-417e-9226-2d987a705dc5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:38.271767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.271678 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:38.648650 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:21:38.648621 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b78432_2478_417e_9226_2d987a705dc5.slice/crio-602d300c8c3c6a1f3867ca04dd1555dd3ebf72ea57be6c53517486ac611953bc WatchSource:0}: Error finding container 602d300c8c3c6a1f3867ca04dd1555dd3ebf72ea57be6c53517486ac611953bc: Status 404 returned error can't find the container with id 602d300c8c3c6a1f3867ca04dd1555dd3ebf72ea57be6c53517486ac611953bc Nov 03 17:21:38.649778 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:38.649752 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm"] Nov 03 17:21:39.525126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:39.525091 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" event={"ID":"b9b78432-2478-417e-9226-2d987a705dc5","Type":"ContainerStarted","Data":"602d300c8c3c6a1f3867ca04dd1555dd3ebf72ea57be6c53517486ac611953bc"} Nov 03 17:21:41.534287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:41.534208 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" event={"ID":"b9b78432-2478-417e-9226-2d987a705dc5","Type":"ContainerStarted","Data":"5780746255feeca4411274d0f4a39bd68b179c68d43b8ae95376d90923996ec9"} Nov 03 17:21:41.534645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:41.534305 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:46.514359 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:46.514327 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:21:46.738629 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:46.738574 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" podStartSLOduration=7.106393244 podStartE2EDuration="9.73856035s" totalImagesPullingTime="2.632167106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:41.658369554 +0000 UTC m=+244.113058577" watchObservedRunningTime="2026-11-03 17:21:46.73856035 +0000 UTC m=+249.193249365" Nov 03 17:21:47.055686 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.055653 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 17:21:47.058899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.058884 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.126640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.126610 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.126777 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.126655 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxq6m\" (UniqueName: \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.167979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.167951 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 17:21:47.227528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.227498 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.227715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.227540 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxq6m\" (UniqueName: \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.227898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.227878 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.295527 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.295502 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxq6m\" (UniqueName: \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\") pod \"kuadrant-operator-controller-manager-678d87766f-wdgfb\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.370605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.370576 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:47.518224 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.518194 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-xbdkc" Nov 03 17:21:47.805502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:47.805474 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 17:21:47.806424 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:21:47.806401 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25335e8_e46d_4757_9cf4_0b36465d0b8e.slice/crio-4d080040a01a3c82aad0e37139441c6db3047b03f7188f98821089e937ef99fe WatchSource:0}: Error finding container 4d080040a01a3c82aad0e37139441c6db3047b03f7188f98821089e937ef99fe: Status 404 returned error can't find the container with id 4d080040a01a3c82aad0e37139441c6db3047b03f7188f98821089e937ef99fe Nov 03 17:21:48.561409 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:48.561375 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" event={"ID":"a25335e8-e46d-4757-9cf4-0b36465d0b8e","Type":"ContainerStarted","Data":"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d"} Nov 03 17:21:48.561798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:48.561432 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" event={"ID":"a25335e8-e46d-4757-9cf4-0b36465d0b8e","Type":"ContainerStarted","Data":"4d080040a01a3c82aad0e37139441c6db3047b03f7188f98821089e937ef99fe"} Nov 03 17:21:48.561798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:48.561491 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:48.605103 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:21:48.605048 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:21:52.539559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:52.539531 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-4lcxm" Nov 03 17:21:52.678219 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:52.678169 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" podStartSLOduration=5.678156067 podStartE2EDuration="5.678156067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:48.698024481 +0000 UTC m=+251.152713498" watchObservedRunningTime="2026-11-03 17:21:52.678156067 +0000 UTC m=+255.132845092" Nov 03 17:21:59.568255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:59.568221 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 17:21:59.804355 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:59.804323 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:21:59.804581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:21:59.804554 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" podUID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" containerName="manager" containerID="cri-o://2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4" gracePeriod=10 Nov 03 17:22:00.645522 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.645491 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:00.648761 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.648744 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.716998 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.716968 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:00.731764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.731727 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2skjc\" (UniqueName: \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.731926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.731780 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.765978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.765946 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 17:22:00.765978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.765967 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-grtmb\"" Nov 03 17:22:00.832462 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.832431 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2skjc\" (UniqueName: \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.832659 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.832472 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.833048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.833030 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.886235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.886201 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:00.897678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.897627 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2skjc\" (UniqueName: \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\") pod \"limitador-limitador-74dfcfb894-pfzv9\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:00.960626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:00.960582 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:01.351439 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:22:01.351407 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6bf3b6_929c_49e0_9ba4_392c12887c11.slice/crio-5959d3f8e676835cd7a3fb5a091eba60f24df96a03792b118c0dd5ec6e267eb1 WatchSource:0}: Error finding container 5959d3f8e676835cd7a3fb5a091eba60f24df96a03792b118c0dd5ec6e267eb1: Status 404 returned error can't find the container with id 5959d3f8e676835cd7a3fb5a091eba60f24df96a03792b118c0dd5ec6e267eb1 Nov 03 17:22:01.353445 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:01.353423 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:01.611372 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:01.611284 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" event={"ID":"0c6bf3b6-929c-49e0-9ba4-392c12887c11","Type":"ContainerStarted","Data":"5959d3f8e676835cd7a3fb5a091eba60f24df96a03792b118c0dd5ec6e267eb1"} Nov 03 17:22:01.947627 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:01.947603 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:22:02.044736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.044699 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\" (UniqueName: \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\") pod \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " Nov 03 17:22:02.044913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.044747 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\") pod \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\" (UID: \"6d233ec7-bec3-41a2-a1e2-aa9c31de4903\") " Nov 03 17:22:02.045198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.045172 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume" pod "6d233ec7-bec3-41a2-a1e2-aa9c31de4903" (UID: "6d233ec7-bec3-41a2-a1e2-aa9c31de4903"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:02.047156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.047137 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks" pod "6d233ec7-bec3-41a2-a1e2-aa9c31de4903" (UID: "6d233ec7-bec3-41a2-a1e2-aa9c31de4903"). InnerVolumeSpecName "kube-api-access-6r9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:02.145446 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.145366 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9ks\" (UniqueName: \"kubernetes.io/projected/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-kube-api-access-6r9ks\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:22:02.145446 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.145398 4885 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d233ec7-bec3-41a2-a1e2-aa9c31de4903-extensions-socket-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:22:02.617084 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.616981 4885 generic.go:356] "Generic (PLEG): container finished" podID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" containerID="2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4" exitCode=0 Nov 03 17:22:02.617084 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.617029 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" Nov 03 17:22:02.617263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.617030 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" event={"ID":"6d233ec7-bec3-41a2-a1e2-aa9c31de4903","Type":"ContainerDied","Data":"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4"} Nov 03 17:22:02.617263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.617149 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb" event={"ID":"6d233ec7-bec3-41a2-a1e2-aa9c31de4903","Type":"ContainerDied","Data":"b34caa3ef75847da4fad6b7e7377bf0742431d292f4e79b9c26309a93f57dc49"} Nov 03 17:22:02.617263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.617174 4885 scope.go:122] "RemoveContainer" containerID="2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4" Nov 03 17:22:02.625377 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.625361 4885 scope.go:122] "RemoveContainer" containerID="2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4" Nov 03 17:22:02.625640 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:22:02.625624 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4\": container with ID starting with 2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4 not found: ID does not exist" containerID="2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4" Nov 03 17:22:02.625677 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.625650 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4"} err="failed to get container status \"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4\": rpc error: code = NotFound desc = could not find container \"2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4\": container with ID starting with 2faf2d7977844f58e631ff109e85d1fb7c6daa7dddc832e4904fc077a49b80f4 not found: ID does not exist" Nov 03 17:22:02.790486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.790455 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:22:02.842167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:02.842122 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-nhfdb"] Nov 03 17:22:03.660509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:03.660475 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" path="/var/lib/kubelet/pods/6d233ec7-bec3-41a2-a1e2-aa9c31de4903/volumes" Nov 03 17:22:04.628776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:04.628743 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" event={"ID":"0c6bf3b6-929c-49e0-9ba4-392c12887c11","Type":"ContainerStarted","Data":"9adf80808541d60957100d9b2b89106fc329eb1e15afd2d33e721df425908b3d"} Nov 03 17:22:04.628984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:04.628898 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:04.753209 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:04.753157 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" podStartSLOduration=2.387730593 podStartE2EDuration="4.753145285s" totalImagesPullingTime="2.365414692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:04.752323806 +0000 UTC m=+267.207012831" watchObservedRunningTime="2026-11-03 17:22:04.753145285 +0000 UTC m=+267.207834309" Nov 03 17:22:11.153919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.153890 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552"] Nov 03 17:22:11.154311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.154195 4885 cpu_manager.go:414] "containerMap: removing container" podUID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" containerName="manager" Nov 03 17:22:11.154311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.154206 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" containerName="manager" Nov 03 17:22:11.154311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.154275 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d233ec7-bec3-41a2-a1e2-aa9c31de4903" containerName="manager" Nov 03 17:22:11.158797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.158776 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.213331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.213305 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-dmxx6\"" Nov 03 17:22:11.218570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.218543 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.218677 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.218598 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsxd\" (UniqueName: \"kubernetes.io/projected/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-kube-api-access-7nsxd\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.218714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.218682 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.218750 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.218729 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.264037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.264009 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 17:22:11.264037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.264012 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 17:22:11.319895 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.319862 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.319910 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.319949 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.319979 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsxd\" (UniqueName: \"kubernetes.io/projected/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-kube-api-access-7nsxd\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.320371 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.320348 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.320428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.320355 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.367886 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.367845 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 17:22:11.369037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.369013 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552"] Nov 03 17:22:11.372045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.372021 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.384690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.384667 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsxd\" (UniqueName: \"kubernetes.io/projected/6b2b23ec-041e-4b09-bc89-fa5e2399b5fc-kube-api-access-7nsxd\") pod \"kserve-module-controller-manager-7566cf5fc8-xj552\" (UID: \"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.469770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.469688 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:11.885314 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:22:11.885283 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2b23ec_041e_4b09_bc89_fa5e2399b5fc.slice/crio-03ebe58f9d1b22c37c1981a4d75241db624351b37948364bccfd7fe5de1216d6 WatchSource:0}: Error finding container 03ebe58f9d1b22c37c1981a4d75241db624351b37948364bccfd7fe5de1216d6: Status 404 returned error can't find the container with id 03ebe58f9d1b22c37c1981a4d75241db624351b37948364bccfd7fe5de1216d6 Nov 03 17:22:11.887172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:11.887153 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552"] Nov 03 17:22:12.660045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:12.660005 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" event={"ID":"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc","Type":"ContainerStarted","Data":"03ebe58f9d1b22c37c1981a4d75241db624351b37948364bccfd7fe5de1216d6"} Nov 03 17:22:15.634806 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:15.634778 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:15.674298 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:15.674268 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" event={"ID":"6b2b23ec-041e-4b09-bc89-fa5e2399b5fc","Type":"ContainerStarted","Data":"655f57f22bc05be1169a9b1363e6745f3de637173e2811947837ce0eb5251bcd"} Nov 03 17:22:15.674448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:15.674326 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:15.875717 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:15.875664 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" podStartSLOduration=1.9901129929999999 podStartE2EDuration="4.875650275s" totalImagesPullingTime="2.885537282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:15.875606219 +0000 UTC m=+278.330295243" watchObservedRunningTime="2026-11-03 17:22:15.875650275 +0000 UTC m=+278.330339291" Nov 03 17:22:18.604574 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:22:18.604537 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:22:21.682401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.682371 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:22:21.684742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.684727 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.746979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.746952 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:22:21.807424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.807389 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6g7\" (UniqueName: \"kubernetes.io/projected/2003487e-5c5d-4661-bfdb-b9b72e729cb6-kube-api-access-sx6g7\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.807614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.807447 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2003487e-5c5d-4661-bfdb-b9b72e729cb6-config-file\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.908582 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.908548 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6g7\" (UniqueName: \"kubernetes.io/projected/2003487e-5c5d-4661-bfdb-b9b72e729cb6-kube-api-access-sx6g7\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.908766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.908605 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2003487e-5c5d-4661-bfdb-b9b72e729cb6-config-file\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.909179 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.909160 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2003487e-5c5d-4661-bfdb-b9b72e729cb6-config-file\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.969936 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.969850 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:22:21.986753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.986727 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6g7\" (UniqueName: \"kubernetes.io/projected/2003487e-5c5d-4661-bfdb-b9b72e729cb6-kube-api-access-sx6g7\") pod \"limitador-limitador-7c9b87d574-sdh8h\" (UID: \"2003487e-5c5d-4661-bfdb-b9b72e729cb6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:21.996602 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:21.996583 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:22.373197 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:22:22.373151 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2003487e_5c5d_4661_bfdb_b9b72e729cb6.slice/crio-9a375c8fb54e441ebe640f82e0a7e620cf544f6cbc0f7233de85c0982625b806 WatchSource:0}: Error finding container 9a375c8fb54e441ebe640f82e0a7e620cf544f6cbc0f7233de85c0982625b806: Status 404 returned error can't find the container with id 9a375c8fb54e441ebe640f82e0a7e620cf544f6cbc0f7233de85c0982625b806 Nov 03 17:22:22.375142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:22.375117 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:22:22.703764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:22.703668 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" event={"ID":"2003487e-5c5d-4661-bfdb-b9b72e729cb6","Type":"ContainerStarted","Data":"6ac436a4c3cd30f6f580753a95652aafc5b5183e7c1aeef83562fdb6f4b56730"} Nov 03 17:22:22.703764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:22.703723 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" event={"ID":"2003487e-5c5d-4661-bfdb-b9b72e729cb6","Type":"ContainerStarted","Data":"9a375c8fb54e441ebe640f82e0a7e620cf544f6cbc0f7233de85c0982625b806"} Nov 03 17:22:22.703764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:22.703766 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:22.831295 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:22.831245 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" podStartSLOduration=1.8312280520000002 podStartE2EDuration="1.831228052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:22:22.830435599 +0000 UTC m=+285.285124624" watchObservedRunningTime="2026-11-03 17:22:22.831228052 +0000 UTC m=+285.285917078" Nov 03 17:22:33.708364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:33.708283 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sdh8h" Nov 03 17:22:33.877330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:33.877298 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:33.877525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:33.877503 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" podUID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" containerName="limitador" containerID="cri-o://9adf80808541d60957100d9b2b89106fc329eb1e15afd2d33e721df425908b3d" gracePeriod=30 Nov 03 17:22:34.754191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.754162 4885 generic.go:356] "Generic (PLEG): container finished" podID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" containerID="9adf80808541d60957100d9b2b89106fc329eb1e15afd2d33e721df425908b3d" exitCode=0 Nov 03 17:22:34.754191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.754200 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" event={"ID":"0c6bf3b6-929c-49e0-9ba4-392c12887c11","Type":"ContainerDied","Data":"9adf80808541d60957100d9b2b89106fc329eb1e15afd2d33e721df425908b3d"} Nov 03 17:22:34.823661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.823636 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:34.917054 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.917023 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\" (UniqueName: \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\") pod \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " Nov 03 17:22:34.917429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.917156 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\" (UniqueName: \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\") pod \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\" (UID: \"0c6bf3b6-929c-49e0-9ba4-392c12887c11\") " Nov 03 17:22:34.917429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.917409 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file" pod "0c6bf3b6-929c-49e0-9ba4-392c12887c11" (UID: "0c6bf3b6-929c-49e0-9ba4-392c12887c11"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:34.919443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:34.919421 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc" pod "0c6bf3b6-929c-49e0-9ba4-392c12887c11" (UID: "0c6bf3b6-929c-49e0-9ba4-392c12887c11"). InnerVolumeSpecName "kube-api-access-2skjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:35.018257 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.018174 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2skjc\" (UniqueName: \"kubernetes.io/projected/0c6bf3b6-929c-49e0-9ba4-392c12887c11-kube-api-access-2skjc\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:22:35.018257 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.018205 4885 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c6bf3b6-929c-49e0-9ba4-392c12887c11-config-file\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:22:35.759470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.759437 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" Nov 03 17:22:35.759470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.759451 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pfzv9" event={"ID":"0c6bf3b6-929c-49e0-9ba4-392c12887c11","Type":"ContainerDied","Data":"5959d3f8e676835cd7a3fb5a091eba60f24df96a03792b118c0dd5ec6e267eb1"} Nov 03 17:22:35.759948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.759515 4885 scope.go:122] "RemoveContainer" containerID="9adf80808541d60957100d9b2b89106fc329eb1e15afd2d33e721df425908b3d" Nov 03 17:22:35.937189 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.937101 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:35.973672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:35.973641 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pfzv9"] Nov 03 17:22:36.680010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:36.679980 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-xj552" Nov 03 17:22:37.661013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:22:37.660983 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" path="/var/lib/kubelet/pods/0c6bf3b6-929c-49e0-9ba4-392c12887c11/volumes" Nov 03 17:22:48.605158 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:22:48.605119 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:23:04.504675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.504644 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:04.505237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.505084 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" containerName="limitador" Nov 03 17:23:04.505237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.505104 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" containerName="limitador" Nov 03 17:23:04.505237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.505181 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6bf3b6-929c-49e0-9ba4-392c12887c11" containerName="limitador" Nov 03 17:23:04.507886 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.507870 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.554463 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.554430 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5wg\" (UniqueName: \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.554668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.554482 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.554668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.554500 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.618337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.618307 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 17:23:04.618526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.618345 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hl2hg\"" Nov 03 17:23:04.618526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.618343 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 17:23:04.618526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.618387 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 17:23:04.620632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.620613 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:04.655042 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.655004 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.655250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.655048 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.655250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.655157 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5wg\" (UniqueName: \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.657764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.657742 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.657914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.657881 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.721350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.721321 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5wg\" (UniqueName: \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\") pod \"maas-controller-58d9f59ffd-qq92d\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:04.819956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:04.819876 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:05.187586 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.187553 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 17:23:05.191517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.191491 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.196107 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:05.196081 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0915e15_8549_4b15_a70e_8ec22dcdff51.slice/crio-4147176397b2ed188696fd75e975bc1bf369c6b0c19889313b561d87b23eeca0 WatchSource:0}: Error finding container 4147176397b2ed188696fd75e975bc1bf369c6b0c19889313b561d87b23eeca0: Status 404 returned error can't find the container with id 4147176397b2ed188696fd75e975bc1bf369c6b0c19889313b561d87b23eeca0 Nov 03 17:23:05.196759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.196737 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:05.197531 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.197517 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:23:05.249962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.249936 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 17:23:05.260749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.260721 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.260880 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.260789 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.260880 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.260851 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxf9h\" (UniqueName: \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.362052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.362016 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxf9h\" (UniqueName: \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.362052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.362061 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.362292 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.362135 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.364722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.364698 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.364835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.364732 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.427898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.427869 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxf9h\" (UniqueName: \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\") pod \"maas-controller-75c4dd589d-rbr8b\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.505572 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.505463 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:05.878168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.878127 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" event={"ID":"c0915e15-8549-4b15-a70e-8ec22dcdff51","Type":"ContainerStarted","Data":"4147176397b2ed188696fd75e975bc1bf369c6b0c19889313b561d87b23eeca0"} Nov 03 17:23:05.902080 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:05.902034 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee922e87_96e2_466a_baf9_082bc2238f2c.slice/crio-ffb68ac9056f552b261cc867415508f32fd60aaae738a1e7bf59a593144cfc91 WatchSource:0}: Error finding container ffb68ac9056f552b261cc867415508f32fd60aaae738a1e7bf59a593144cfc91: Status 404 returned error can't find the container with id ffb68ac9056f552b261cc867415508f32fd60aaae738a1e7bf59a593144cfc91 Nov 03 17:23:05.904550 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:05.904432 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 17:23:06.883316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:06.883284 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" event={"ID":"ee922e87-96e2-466a-baf9-082bc2238f2c","Type":"ContainerStarted","Data":"ffb68ac9056f552b261cc867415508f32fd60aaae738a1e7bf59a593144cfc91"} Nov 03 17:23:12.910181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:12.910055 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" event={"ID":"c0915e15-8549-4b15-a70e-8ec22dcdff51","Type":"ContainerStarted","Data":"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5"} Nov 03 17:23:12.910558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:12.910250 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:12.911761 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:12.911734 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" event={"ID":"ee922e87-96e2-466a-baf9-082bc2238f2c","Type":"ContainerStarted","Data":"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3"} Nov 03 17:23:12.911885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:12.911811 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:13.034464 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:13.034418 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" podStartSLOduration=1.6119741520000002 podStartE2EDuration="9.034404869s" totalImagesPullingTime="7.422430717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:13.033563593 +0000 UTC m=+335.488252620" watchObservedRunningTime="2026-11-03 17:23:13.034404869 +0000 UTC m=+335.489093894" Nov 03 17:23:14.028541 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.028481 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" podStartSLOduration=2.65718225 podStartE2EDuration="9.028459225s" totalImagesPullingTime="6.371276975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:13.157237865 +0000 UTC m=+335.611926890" watchObservedRunningTime="2026-11-03 17:23:14.028459225 +0000 UTC m=+336.483148250" Nov 03 17:23:14.029796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.029775 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt"] Nov 03 17:23:14.033804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.033785 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.120798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.120766 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt"] Nov 03 17:23:14.139863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.139828 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.139863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.139859 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91117bda-a060-42e8-a82f-466590963e91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.140043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.139889 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.140043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.139989 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.140043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.140011 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91117bda-a060-42e8-a82f-466590963e91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.140043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.140028 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.140043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.140043 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtwc\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-kube-api-access-8wtwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240661 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240704 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91117bda-a060-42e8-a82f-466590963e91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240728 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240751 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtwc\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-kube-api-access-8wtwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240810 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240833 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91117bda-a060-42e8-a82f-466590963e91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.240919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.240872 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.241623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.241599 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91117bda-a060-42e8-a82f-466590963e91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.243558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.243492 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.243558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.243522 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.243738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.243714 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91117bda-a060-42e8-a82f-466590963e91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.243861 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.243845 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91117bda-a060-42e8-a82f-466590963e91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.304258 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.304178 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.304468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.304447 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtwc\" (UniqueName: \"kubernetes.io/projected/91117bda-a060-42e8-a82f-466590963e91-kube-api-access-8wtwc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvgtt\" (UID: \"91117bda-a060-42e8-a82f-466590963e91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.350657 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.350629 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:14.774562 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:14.774532 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91117bda_a060_42e8_a82f_466590963e91.slice/crio-c9be8d9e0154130e8367f5a6cae916c5bef09dd5b1bc61398d124a61fb32daa7 WatchSource:0}: Error finding container c9be8d9e0154130e8367f5a6cae916c5bef09dd5b1bc61398d124a61fb32daa7: Status 404 returned error can't find the container with id c9be8d9e0154130e8367f5a6cae916c5bef09dd5b1bc61398d124a61fb32daa7 Nov 03 17:23:14.775747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.775720 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt"] Nov 03 17:23:14.776646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.776616 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:23:14.776742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.776675 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:23:14.922000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.921961 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" event={"ID":"91117bda-a060-42e8-a82f-466590963e91","Type":"ContainerStarted","Data":"932c0f282bed039146ebe6772693796c2672abc28bf853662d1e0999554584a8"} Nov 03 17:23:14.922173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.922020 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" event={"ID":"91117bda-a060-42e8-a82f-466590963e91","Type":"ContainerStarted","Data":"c9be8d9e0154130e8367f5a6cae916c5bef09dd5b1bc61398d124a61fb32daa7"} Nov 03 17:23:14.922173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:14.922043 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:15.047334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:15.047237 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" podStartSLOduration=2.047222644 podStartE2EDuration="2.047222644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:15.046028965 +0000 UTC m=+337.500717989" watchObservedRunningTime="2026-11-03 17:23:15.047222644 +0000 UTC m=+337.501911668" Nov 03 17:23:15.928851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:15.928819 4885 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rvgtt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:23:15.929003 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:15.928865 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" podUID="91117bda-a060-42e8-a82f-466590963e91" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:23:18.604623 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:23:18.604582 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:23:18.928030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:18.928005 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvgtt" Nov 03 17:23:19.166952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.166913 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:23:19.167271 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.167218 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" containerID="cri-o://3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4" gracePeriod=30 Nov 03 17:23:19.299665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.299549 4885 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hgvzt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.13:8080/ready\": dial tcp 10.134.0.13:8080: connect: connection refused" start-of-body= Nov 03 17:23:19.299955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.299897 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.13:8080/ready\": dial tcp 10.134.0.13:8080: connect: connection refused" Nov 03 17:23:19.417674 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.417651 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:23:19.492026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.491952 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.491984 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492036 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492080 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492120 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492148 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492254 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\") pod \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\" (UID: \"647b58ea-93f7-4b5e-bd2c-872ccde7695c\") " Nov 03 17:23:19.492469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492392 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:23:19.492579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.492508 4885 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-ca-configmap\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.495244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495181 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:23:19.495377 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495254 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:19.495511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495485 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:19.495623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495562 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:19.495623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495583 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:19.495623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.495581 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp" pod "647b58ea-93f7-4b5e-bd2c-872ccde7695c" (UID: "647b58ea-93f7-4b5e-bd2c-872ccde7695c"). InnerVolumeSpecName "kube-api-access-dzgmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:19.593768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593737 4885 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-csr-dns-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.593768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593766 4885 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-kubeconfig\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.593962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593779 4885 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/647b58ea-93f7-4b5e-bd2c-872ccde7695c-local-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.593962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593791 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.593962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593804 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgmp\" (UniqueName: \"kubernetes.io/projected/647b58ea-93f7-4b5e-bd2c-872ccde7695c-kube-api-access-dzgmp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.593962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.593818 4885 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/647b58ea-93f7-4b5e-bd2c-872ccde7695c-cacerts\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:19.756561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.756495 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-r7rw5"] Nov 03 17:23:19.756922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.756800 4885 cpu_manager.go:414] "containerMap: removing container" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" Nov 03 17:23:19.756922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.756811 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" Nov 03 17:23:19.756922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.756875 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerName="discovery" Nov 03 17:23:19.759959 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.759938 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:19.814593 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.814570 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 17:23:19.814754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.814604 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z44wt\"" Nov 03 17:23:19.823556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.823531 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-r7rw5"] Nov 03 17:23:19.871104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.871055 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:23:19.874302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.874284 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:19.895700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.895677 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgxs\" (UniqueName: \"kubernetes.io/projected/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-kube-api-access-2wgxs\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:19.895794 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.895708 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-cert\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:19.930187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.930164 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 17:23:19.930314 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.930249 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-htqlr\"" Nov 03 17:23:19.930914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.930894 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:23:19.943694 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.943672 4885 generic.go:356] "Generic (PLEG): container finished" podID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" containerID="3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4" exitCode=0 Nov 03 17:23:19.943796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.943700 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" event={"ID":"647b58ea-93f7-4b5e-bd2c-872ccde7695c","Type":"ContainerDied","Data":"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4"} Nov 03 17:23:19.943796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.943715 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" Nov 03 17:23:19.943796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.943728 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt" event={"ID":"647b58ea-93f7-4b5e-bd2c-872ccde7695c","Type":"ContainerDied","Data":"1c48bc9074837439f9aa3a0973db8fba79e45178fdbc3d1fefd70a2444c2a98f"} Nov 03 17:23:19.943796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.943746 4885 scope.go:122] "RemoveContainer" containerID="3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4" Nov 03 17:23:19.951769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.951702 4885 scope.go:122] "RemoveContainer" containerID="3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4" Nov 03 17:23:19.952016 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:23:19.951998 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4\": container with ID starting with 3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4 not found: ID does not exist" containerID="3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4" Nov 03 17:23:19.952124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.952027 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4"} err="failed to get container status \"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4\": rpc error: code = NotFound desc = could not find container \"3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4\": container with ID starting with 3fb5047c3ee568d9500eb3275c4f644a279764ca91e05c7dc4d8a44b021e36b4 not found: ID does not exist" Nov 03 17:23:19.996387 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.996358 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgxs\" (UniqueName: \"kubernetes.io/projected/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-kube-api-access-2wgxs\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:19.996511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.996399 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-cert\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:19.996511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.996451 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:19.996581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.996549 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9vn\" (UniqueName: \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:19.998904 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:19.998887 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-cert\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:20.066183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.066119 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgxs\" (UniqueName: \"kubernetes.io/projected/19e0a745-46bf-4cb6-a26b-bc8c643aa60d-kube-api-access-2wgxs\") pod \"kserve-controller-manager-776cc66d47-r7rw5\" (UID: \"19e0a745-46bf-4cb6-a26b-bc8c643aa60d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:20.071947 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.071923 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:20.097554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.097528 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:20.097679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.097604 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9vn\" (UniqueName: \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:20.100177 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.100151 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:20.123855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.123824 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:23:20.133427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.133405 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hgvzt"] Nov 03 17:23:20.160182 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.160154 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9vn\" (UniqueName: \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\") pod \"llmisvc-controller-manager-869d5dcc5d-w5czh\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:20.186049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.186017 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:20.444008 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:20.443966 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0a745_46bf_4cb6_a26b_bc8c643aa60d.slice/crio-11802c9290a7dd9cc8a35c92fee81b4cb06c09297ab9d36af797d5e6bb1a9a0d WatchSource:0}: Error finding container 11802c9290a7dd9cc8a35c92fee81b4cb06c09297ab9d36af797d5e6bb1a9a0d: Status 404 returned error can't find the container with id 11802c9290a7dd9cc8a35c92fee81b4cb06c09297ab9d36af797d5e6bb1a9a0d Nov 03 17:23:20.445631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.445601 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-r7rw5"] Nov 03 17:23:20.550846 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:20.550816 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33a64a28_7810_4994_a285_a93e4dfddb26.slice/crio-517e5bfe42ee7b6cc3296d3ccd1b6e9586c3097b28faa87c3ce198aa1e82437b WatchSource:0}: Error finding container 517e5bfe42ee7b6cc3296d3ccd1b6e9586c3097b28faa87c3ce198aa1e82437b: Status 404 returned error can't find the container with id 517e5bfe42ee7b6cc3296d3ccd1b6e9586c3097b28faa87c3ce198aa1e82437b Nov 03 17:23:20.553090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.553046 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:23:20.949039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.948995 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" event={"ID":"19e0a745-46bf-4cb6-a26b-bc8c643aa60d","Type":"ContainerStarted","Data":"11802c9290a7dd9cc8a35c92fee81b4cb06c09297ab9d36af797d5e6bb1a9a0d"} Nov 03 17:23:20.950163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:20.950141 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" event={"ID":"33a64a28-7810-4994-a285-a93e4dfddb26","Type":"ContainerStarted","Data":"517e5bfe42ee7b6cc3296d3ccd1b6e9586c3097b28faa87c3ce198aa1e82437b"} Nov 03 17:23:21.662217 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:21.662182 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647b58ea-93f7-4b5e-bd2c-872ccde7695c" path="/var/lib/kubelet/pods/647b58ea-93f7-4b5e-bd2c-872ccde7695c/volumes" Nov 03 17:23:23.918576 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:23.918541 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:23.919034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:23.918927 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 17:23:23.966754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:23.966262 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" event={"ID":"33a64a28-7810-4994-a285-a93e4dfddb26","Type":"ContainerStarted","Data":"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a"} Nov 03 17:23:23.966754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:23.966412 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:24.167145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.167093 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" podStartSLOduration=1.96418083 podStartE2EDuration="5.167056271s" totalImagesPullingTime="3.202875441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:24.164536899 +0000 UTC m=+346.619225924" watchObservedRunningTime="2026-11-03 17:23:24.167056271 +0000 UTC m=+346.621745295" Nov 03 17:23:24.317521 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.317491 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:24.317718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.317696 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" podUID="c0915e15-8549-4b15-a70e-8ec22dcdff51" containerName="manager" containerID="cri-o://6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5" gracePeriod=30 Nov 03 17:23:24.564399 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.564375 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:24.640160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.640129 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\" (UniqueName: \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\") pod \"c0915e15-8549-4b15-a70e-8ec22dcdff51\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " Nov 03 17:23:24.640333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.640216 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\") pod \"c0915e15-8549-4b15-a70e-8ec22dcdff51\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " Nov 03 17:23:24.640333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.640252 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\") pod \"c0915e15-8549-4b15-a70e-8ec22dcdff51\" (UID: \"c0915e15-8549-4b15-a70e-8ec22dcdff51\") " Nov 03 17:23:24.642463 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.642437 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs" pod "c0915e15-8549-4b15-a70e-8ec22dcdff51" (UID: "c0915e15-8549-4b15-a70e-8ec22dcdff51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:24.642584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.642539 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert" pod "c0915e15-8549-4b15-a70e-8ec22dcdff51" (UID: "c0915e15-8549-4b15-a70e-8ec22dcdff51"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:24.642584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.642561 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg" pod "c0915e15-8549-4b15-a70e-8ec22dcdff51" (UID: "c0915e15-8549-4b15-a70e-8ec22dcdff51"). InnerVolumeSpecName "kube-api-access-cg5wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:24.741720 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.741676 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5wg\" (UniqueName: \"kubernetes.io/projected/c0915e15-8549-4b15-a70e-8ec22dcdff51-kube-api-access-cg5wg\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:24.741720 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.741715 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:24.741720 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.741725 4885 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0915e15-8549-4b15-a70e-8ec22dcdff51-webhook-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:23:24.971938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.971846 4885 generic.go:356] "Generic (PLEG): container finished" podID="c0915e15-8549-4b15-a70e-8ec22dcdff51" containerID="6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5" exitCode=0 Nov 03 17:23:24.971938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.971910 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" Nov 03 17:23:24.971938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.971923 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" event={"ID":"c0915e15-8549-4b15-a70e-8ec22dcdff51","Type":"ContainerDied","Data":"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5"} Nov 03 17:23:24.972544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.971974 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58d9f59ffd-qq92d" event={"ID":"c0915e15-8549-4b15-a70e-8ec22dcdff51","Type":"ContainerDied","Data":"4147176397b2ed188696fd75e975bc1bf369c6b0c19889313b561d87b23eeca0"} Nov 03 17:23:24.972544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.971997 4885 scope.go:122] "RemoveContainer" containerID="6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5" Nov 03 17:23:24.973597 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.973571 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" event={"ID":"19e0a745-46bf-4cb6-a26b-bc8c643aa60d","Type":"ContainerStarted","Data":"d2e560e431ee11737ae3bba1428b814bc1a68b171ea8001c9dcd80e89f9c851b"} Nov 03 17:23:24.973763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.973744 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:24.981188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.981149 4885 scope.go:122] "RemoveContainer" containerID="6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5" Nov 03 17:23:24.981515 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:23:24.981492 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5\": container with ID starting with 6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5 not found: ID does not exist" containerID="6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5" Nov 03 17:23:24.981579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:24.981523 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5"} err="failed to get container status \"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5\": rpc error: code = NotFound desc = could not find container \"6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5\": container with ID starting with 6f6352a6f8f92b5f2f30c1685193def20429414723d5657871db421565272eb5 not found: ID does not exist" Nov 03 17:23:25.139843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:25.139811 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:25.142277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:25.142244 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58d9f59ffd-qq92d"] Nov 03 17:23:25.261192 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:25.261095 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" podStartSLOduration=2.633562421 podStartE2EDuration="6.261054028s" totalImagesPullingTime="3.627491607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:25.260261994 +0000 UTC m=+347.714951037" watchObservedRunningTime="2026-11-03 17:23:25.261054028 +0000 UTC m=+347.715743052" Nov 03 17:23:25.661033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:25.660992 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0915e15-8549-4b15-a70e-8ec22dcdff51" path="/var/lib/kubelet/pods/c0915e15-8549-4b15-a70e-8ec22dcdff51/volumes" Nov 03 17:23:48.606115 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:23:48.606051 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:23:49.569139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.569105 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:23:49.569435 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.569423 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c0915e15-8549-4b15-a70e-8ec22dcdff51" containerName="manager" Nov 03 17:23:49.569476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.569436 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0915e15-8549-4b15-a70e-8ec22dcdff51" containerName="manager" Nov 03 17:23:49.569517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.569498 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0915e15-8549-4b15-a70e-8ec22dcdff51" containerName="manager" Nov 03 17:23:49.574912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.574897 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.632745 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.632715 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:23:49.648170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.648146 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.648289 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.648181 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6shq\" (UniqueName: \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.648289 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.648272 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.648368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.648298 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.683988 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.683958 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:23:49.684194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.684168 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:23:49.684302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.684246 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:23:49.684302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.684283 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 17:23:49.685446 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.685424 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:23:49.749018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.748993 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.749018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.749026 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.749200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.749110 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.749200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.749141 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6shq\" (UniqueName: \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.749800 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.749754 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.751655 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.751634 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.751762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.751659 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.814723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.814701 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6shq\" (UniqueName: \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\") pod \"maas-api-7fb777944c-w4jd8\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:49.886682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:49.886656 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:50.250989 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:23:50.250964 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fb5afe_a6a7_4057_90f6_a126d37c47cb.slice/crio-4259ede656d1bf233c40487213bd2f9beeb5506e9d01a0d75b3c3ad23d094535 WatchSource:0}: Error finding container 4259ede656d1bf233c40487213bd2f9beeb5506e9d01a0d75b3c3ad23d094535: Status 404 returned error can't find the container with id 4259ede656d1bf233c40487213bd2f9beeb5506e9d01a0d75b3c3ad23d094535 Nov 03 17:23:50.252912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:50.252879 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:23:51.084228 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:51.084194 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" event={"ID":"41fb5afe-a6a7-4057-90f6-a126d37c47cb","Type":"ContainerStarted","Data":"4259ede656d1bf233c40487213bd2f9beeb5506e9d01a0d75b3c3ad23d094535"} Nov 03 17:23:54.704235 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:54.704203 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:23:54.976353 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:54.976275 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:23:55.985368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:55.985338 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-r7rw5" Nov 03 17:23:57.110140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:57.110106 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" event={"ID":"41fb5afe-a6a7-4057-90f6-a126d37c47cb","Type":"ContainerStarted","Data":"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210"} Nov 03 17:23:57.110518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:57.110199 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:23:57.289974 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:23:57.289864 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" podStartSLOduration=1.557986337 podStartE2EDuration="8.289850568s" totalImagesPullingTime="6.731864231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:57.28974005 +0000 UTC m=+379.744429075" watchObservedRunningTime="2026-11-03 17:23:57.289850568 +0000 UTC m=+379.744539595" Nov 03 17:24:03.120562 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:03.120482 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:24:14.493425 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.493395 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:24:14.493917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.493673 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" podUID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" containerName="maas-api" containerID="cri-o://8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210" gracePeriod=30 Nov 03 17:24:14.734003 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.733980 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:24:14.857334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.857257 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\") pod \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " Nov 03 17:24:14.857334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.857313 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\") pod \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " Nov 03 17:24:14.857566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.857356 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\") pod \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " Nov 03 17:24:14.857566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.857425 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\" (UniqueName: \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\") pod \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\" (UID: \"41fb5afe-a6a7-4057-90f6-a126d37c47cb\") " Nov 03 17:24:14.857926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.857831 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca" pod "41fb5afe-a6a7-4057-90f6-a126d37c47cb" (UID: "41fb5afe-a6a7-4057-90f6-a126d37c47cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:14.860236 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.860207 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs" pod "41fb5afe-a6a7-4057-90f6-a126d37c47cb" (UID: "41fb5afe-a6a7-4057-90f6-a126d37c47cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:14.860437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.860406 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls" pod "41fb5afe-a6a7-4057-90f6-a126d37c47cb" (UID: "41fb5afe-a6a7-4057-90f6-a126d37c47cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:14.860437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.860453 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq" pod "41fb5afe-a6a7-4057-90f6-a126d37c47cb" (UID: "41fb5afe-a6a7-4057-90f6-a126d37c47cb"). InnerVolumeSpecName "kube-api-access-k6shq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:14.958944 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.958901 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6shq\" (UniqueName: \"kubernetes.io/projected/41fb5afe-a6a7-4057-90f6-a126d37c47cb-kube-api-access-k6shq\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:14.958944 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.958939 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:14.958944 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.958951 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41fb5afe-a6a7-4057-90f6-a126d37c47cb-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:14.958944 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:14.958960 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41fb5afe-a6a7-4057-90f6-a126d37c47cb-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:24:15.184610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.184577 4885 generic.go:356] "Generic (PLEG): container finished" podID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" containerID="8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210" exitCode=0 Nov 03 17:24:15.184751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.184615 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" event={"ID":"41fb5afe-a6a7-4057-90f6-a126d37c47cb","Type":"ContainerDied","Data":"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210"} Nov 03 17:24:15.184751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.184628 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" Nov 03 17:24:15.184751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.184651 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8" event={"ID":"41fb5afe-a6a7-4057-90f6-a126d37c47cb","Type":"ContainerDied","Data":"4259ede656d1bf233c40487213bd2f9beeb5506e9d01a0d75b3c3ad23d094535"} Nov 03 17:24:15.184751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.184673 4885 scope.go:122] "RemoveContainer" containerID="8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210" Nov 03 17:24:15.193421 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.193405 4885 scope.go:122] "RemoveContainer" containerID="8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210" Nov 03 17:24:15.193694 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:24:15.193677 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210\": container with ID starting with 8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210 not found: ID does not exist" containerID="8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210" Nov 03 17:24:15.193745 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.193704 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210"} err="failed to get container status \"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210\": rpc error: code = NotFound desc = could not find container \"8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210\": container with ID starting with 8c40ea9a17d606e77038a117f9802130295b3b91eaf086cbf9492d97e1967210 not found: ID does not exist" Nov 03 17:24:15.364789 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.364760 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:24:15.375479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.375446 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fb777944c-w4jd8"] Nov 03 17:24:15.663852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:15.663814 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" path="/var/lib/kubelet/pods/41fb5afe-a6a7-4057-90f6-a126d37c47cb/volumes" Nov 03 17:24:18.604626 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:24:18.604586 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:24:28.659897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.659858 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v"] Nov 03 17:24:28.660448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.660337 4885 cpu_manager.go:414] "containerMap: removing container" podUID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" containerName="maas-api" Nov 03 17:24:28.660448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.660356 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" containerName="maas-api" Nov 03 17:24:28.660448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.660439 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="41fb5afe-a6a7-4057-90f6-a126d37c47cb" containerName="maas-api" Nov 03 17:24:28.664646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.664623 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.718399 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.718369 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v"] Nov 03 17:24:28.779613 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.779582 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbbf0df-7541-4486-bc3c-861bea91ef4e-cert\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.779860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.779839 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z27g\" (UniqueName: \"kubernetes.io/projected/7fbbf0df-7541-4486-bc3c-861bea91ef4e-kube-api-access-8z27g\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.880533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.880498 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z27g\" (UniqueName: \"kubernetes.io/projected/7fbbf0df-7541-4486-bc3c-861bea91ef4e-kube-api-access-8z27g\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.880697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.880560 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbbf0df-7541-4486-bc3c-861bea91ef4e-cert\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.883164 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.883135 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fbbf0df-7541-4486-bc3c-861bea91ef4e-cert\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.946166 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.946090 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z27g\" (UniqueName: \"kubernetes.io/projected/7fbbf0df-7541-4486-bc3c-861bea91ef4e-kube-api-access-8z27g\") pod \"llmisvc-controller-manager-6466c554d8-hqg4v\" (UID: \"7fbbf0df-7541-4486-bc3c-861bea91ef4e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:28.976957 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:28.976922 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:29.358779 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:24:29.358752 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbbf0df_7541_4486_bc3c_861bea91ef4e.slice/crio-131e94e36c8e5f96149583ffb4f09c5ad7164fae7d0a7ed53305c60210ee6653 WatchSource:0}: Error finding container 131e94e36c8e5f96149583ffb4f09c5ad7164fae7d0a7ed53305c60210ee6653: Status 404 returned error can't find the container with id 131e94e36c8e5f96149583ffb4f09c5ad7164fae7d0a7ed53305c60210ee6653 Nov 03 17:24:29.361784 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:29.361761 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v"] Nov 03 17:24:30.247418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:30.247387 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" event={"ID":"7fbbf0df-7541-4486-bc3c-861bea91ef4e","Type":"ContainerStarted","Data":"450e118c7d665b25aa733488cec73ab3ee87fcee80540acfcbe0dd5d30196b61"} Nov 03 17:24:30.247796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:30.247457 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" event={"ID":"7fbbf0df-7541-4486-bc3c-861bea91ef4e","Type":"ContainerStarted","Data":"131e94e36c8e5f96149583ffb4f09c5ad7164fae7d0a7ed53305c60210ee6653"} Nov 03 17:24:30.247796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:30.247499 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:24:30.369392 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:24:30.369343 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" podStartSLOduration=1.642565218 podStartE2EDuration="2.369329717s" totalImagesPullingTime="726.764499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:30.367833061 +0000 UTC m=+412.822522084" watchObservedRunningTime="2026-11-03 17:24:30.369329717 +0000 UTC m=+412.824018741" Nov 03 17:24:48.605499 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:24:48.605464 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:25:01.254799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.254771 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-hqg4v" Nov 03 17:25:01.433666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.433627 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:25:01.433885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.433862 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" podUID="33a64a28-7810-4994-a285-a93e4dfddb26" containerName="manager" containerID="cri-o://14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a" gracePeriod=30 Nov 03 17:25:01.675265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.675234 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:25:01.727573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.727544 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\" (UniqueName: \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\") pod \"33a64a28-7810-4994-a285-a93e4dfddb26\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " Nov 03 17:25:01.727719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.727610 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\" (UniqueName: \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\") pod \"33a64a28-7810-4994-a285-a93e4dfddb26\" (UID: \"33a64a28-7810-4994-a285-a93e4dfddb26\") " Nov 03 17:25:01.729713 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.729686 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn" pod "33a64a28-7810-4994-a285-a93e4dfddb26" (UID: "33a64a28-7810-4994-a285-a93e4dfddb26"). InnerVolumeSpecName "kube-api-access-mr9vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:01.729810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.729765 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert" pod "33a64a28-7810-4994-a285-a93e4dfddb26" (UID: "33a64a28-7810-4994-a285-a93e4dfddb26"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:01.828892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.828817 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr9vn\" (UniqueName: \"kubernetes.io/projected/33a64a28-7810-4994-a285-a93e4dfddb26-kube-api-access-mr9vn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:01.828892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:01.828843 4885 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33a64a28-7810-4994-a285-a93e4dfddb26-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:25:02.375809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.375771 4885 generic.go:356] "Generic (PLEG): container finished" podID="33a64a28-7810-4994-a285-a93e4dfddb26" containerID="14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a" exitCode=0 Nov 03 17:25:02.376500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.375822 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" Nov 03 17:25:02.376500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.375844 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" event={"ID":"33a64a28-7810-4994-a285-a93e4dfddb26","Type":"ContainerDied","Data":"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a"} Nov 03 17:25:02.376500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.375890 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh" event={"ID":"33a64a28-7810-4994-a285-a93e4dfddb26","Type":"ContainerDied","Data":"517e5bfe42ee7b6cc3296d3ccd1b6e9586c3097b28faa87c3ce198aa1e82437b"} Nov 03 17:25:02.376500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.375912 4885 scope.go:122] "RemoveContainer" containerID="14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a" Nov 03 17:25:02.384562 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.384542 4885 scope.go:122] "RemoveContainer" containerID="14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a" Nov 03 17:25:02.384823 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:25:02.384807 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a\": container with ID starting with 14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a not found: ID does not exist" containerID="14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a" Nov 03 17:25:02.384859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.384833 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a"} err="failed to get container status \"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a\": rpc error: code = NotFound desc = could not find container \"14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a\": container with ID starting with 14426499d6650a7ba60e4ebba5f64b1840a20ea10f11486f414aa79e1cfaeb1a not found: ID does not exist" Nov 03 17:25:02.544613 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.544585 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:25:02.599687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:02.596490 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w5czh"] Nov 03 17:25:03.661168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:03.661135 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a64a28-7810-4994-a285-a93e4dfddb26" path="/var/lib/kubelet/pods/33a64a28-7810-4994-a285-a93e4dfddb26/volumes" Nov 03 17:25:18.604247 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:25:18.604209 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:25:48.604499 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:25:48.604436 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:25:53.577354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:25:53.577314 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:26:05.228741 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:05.228703 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:26:18.604714 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:26:18.604675 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:26:19.729049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:19.729017 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:26:28.379940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.379906 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:26:28.380345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.380225 4885 cpu_manager.go:414] "containerMap: removing container" podUID="33a64a28-7810-4994-a285-a93e4dfddb26" containerName="manager" Nov 03 17:26:28.380345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.380237 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a64a28-7810-4994-a285-a93e4dfddb26" containerName="manager" Nov 03 17:26:28.380345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.380305 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a64a28-7810-4994-a285-a93e4dfddb26" containerName="manager" Nov 03 17:26:28.383318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.383303 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421473 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421505 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421732 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421527 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421732 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421600 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wftl\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421732 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421695 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421735 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421766 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421802 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.421846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.421832 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.437747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.437721 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\"" Nov 03 17:26:28.509559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.509513 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:26:28.522428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522407 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522443 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522459 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522474 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522611 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522648 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522666 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522691 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wftl\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522785 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.522915 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.522879 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.523039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.523014 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.523310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.523290 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.523502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.523481 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.523546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.523519 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.524971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.524955 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.525490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.525467 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.586433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.586404 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.586551 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.586453 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wftl\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\") pod \"maas-default-gateway-openshift-default-58c6585447-rt6t4\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:28.695490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:28.695403 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:29.109819 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:26:29.109792 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd69a74_705a_4c8e_bf13_ff73aec9f58f.slice/crio-53735f91125ceb666a51d2c6f5e639c818add40a119e49ab2d470eb5bfe6f119 WatchSource:0}: Error finding container 53735f91125ceb666a51d2c6f5e639c818add40a119e49ab2d470eb5bfe6f119: Status 404 returned error can't find the container with id 53735f91125ceb666a51d2c6f5e639c818add40a119e49ab2d470eb5bfe6f119 Nov 03 17:26:29.116612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:29.112103 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:26:29.735280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:29.735248 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" event={"ID":"7cd69a74-705a-4c8e-bf13-ff73aec9f58f","Type":"ContainerStarted","Data":"53735f91125ceb666a51d2c6f5e639c818add40a119e49ab2d470eb5bfe6f119"} Nov 03 17:26:34.757788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:34.757756 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:34.758186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:34.757835 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:34.758186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:34.757882 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:35.762242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:35.762198 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" event={"ID":"7cd69a74-705a-4c8e-bf13-ff73aec9f58f","Type":"ContainerStarted","Data":"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d"} Nov 03 17:26:35.939185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:35.939135 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podStartSLOduration=2.299790772 podStartE2EDuration="7.939119812s" totalImagesPullingTime="5.63932904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:35.937979353 +0000 UTC m=+538.392668380" watchObservedRunningTime="2026-11-03 17:26:35.939119812 +0000 UTC m=+538.393808832" Nov 03 17:26:36.696247 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:36.696215 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:36.697448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:36.697419 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:36.697594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:36.697469 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:37.696740 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:37.696707 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:37.697216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:37.696758 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:38.696133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:38.696094 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:38.698174 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:38.696158 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:39.696771 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:39.696736 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:39.696951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:39.696785 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:40.695731 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:40.695699 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:40.696133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:40.695749 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:41.695977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:41.695942 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Nov 03 17:26:41.696476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:41.696014 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Nov 03 17:26:43.486473 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.486441 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:26:43.489869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.489847 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.600187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.600147 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:26:43.682306 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682272 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682306 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682317 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682341 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682398 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682558 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682596 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682943 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682670 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682943 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682726 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.682943 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.682761 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.696374 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.696345 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:43.696559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.696458 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:26:43.696559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.696517 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:26:43.698006 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.697987 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:43.783940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.783839 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.783940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.783895 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.783951 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.783986 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784009 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784124 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784240 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784270 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784325 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784629 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784608 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784813 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784757 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784847 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784923 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.784979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.784936 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.786391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.786372 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.786581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.786565 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.801787 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.801770 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:26:43.850945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.850915 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:43.851133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:43.851007 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\") pod \"maas-default-gateway-openshift-default-58c6585447-xv8cx\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:44.104948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.104868 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:44.519639 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:26:44.519611 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619c15b7_5793_4589_81b5_1a92f2c64741.slice/crio-1034dd1525a64dbeb315a05a400645ac1d30983d006e0480c939f0ed0213c3e5 WatchSource:0}: Error finding container 1034dd1525a64dbeb315a05a400645ac1d30983d006e0480c939f0ed0213c3e5: Status 404 returned error can't find the container with id 1034dd1525a64dbeb315a05a400645ac1d30983d006e0480c939f0ed0213c3e5 Nov 03 17:26:44.521706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.521680 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:26:44.521877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.521850 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:44.521937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.521909 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:44.521976 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.521936 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:26:44.806986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.806897 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" event={"ID":"619c15b7-5793-4589-81b5-1a92f2c64741","Type":"ContainerStarted","Data":"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d"} Nov 03 17:26:44.807171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.806988 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" event={"ID":"619c15b7-5793-4589-81b5-1a92f2c64741","Type":"ContainerStarted","Data":"1034dd1525a64dbeb315a05a400645ac1d30983d006e0480c939f0ed0213c3e5"} Nov 03 17:26:44.939604 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:44.939555 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podStartSLOduration=1.939539509 podStartE2EDuration="1.939539509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:44.939455713 +0000 UTC m=+547.394144738" watchObservedRunningTime="2026-11-03 17:26:44.939539509 +0000 UTC m=+547.394228533" Nov 03 17:26:45.115155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:45.114729 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:45.115155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:45.114858 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:45.115155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:45.114920 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:46.119445 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:46.119410 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:46.119994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:46.119966 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:47.114477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:47.114440 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:47.114770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:47.114742 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:48.110092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:48.106427 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:48.110092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:48.106523 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:48.631292 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:26:48.631257 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:26:49.107571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:49.107474 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:49.107821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:49.107789 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:50.118094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:50.112410 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:50.118094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:50.112488 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:51.106213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:51.106172 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:51.106376 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:51.106240 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:52.106245 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:52.106210 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Nov 03 17:26:52.106724 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:52.106269 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Nov 03 17:26:53.576396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:53.576367 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:53.576891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:53.576677 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:26:53.577468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:26:53.577443 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:27:18.606769 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:27:18.606682 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:27:38.194679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.194648 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:27:38.197213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.197191 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.305270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.305239 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8fc5ec-serving-cert\"" Nov 03 17:27:38.305494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.305258 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:27:38.305494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.305267 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:27:38.305699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.305268 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:27:38.305699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.305360 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:27:38.307158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.307137 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:27:38.388406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.388364 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9nl\" (UniqueName: \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.388623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.388454 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.388623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.388496 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.388623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.388581 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.490103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.490005 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9nl\" (UniqueName: \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.490103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.490052 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.490103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.490093 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.490382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.490126 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.490672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.490638 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.492743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.492717 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.492830 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.492762 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.553049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.553019 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9nl\" (UniqueName: \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\") pod \"maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:38.808130 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:38.808030 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:39.180555 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:27:39.180524 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220de158_6794_4079_ac22_7969ed4a4699.slice/crio-dea123ecaa8ba545b3975f8c10c350249693b4553f774db6247fa65cc0d43cb9 WatchSource:0}: Error finding container dea123ecaa8ba545b3975f8c10c350249693b4553f774db6247fa65cc0d43cb9: Status 404 returned error can't find the container with id dea123ecaa8ba545b3975f8c10c350249693b4553f774db6247fa65cc0d43cb9 Nov 03 17:27:39.182274 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:39.182253 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:27:40.045057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:40.045022 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" event={"ID":"220de158-6794-4079-ac22-7969ed4a4699","Type":"ContainerStarted","Data":"dea123ecaa8ba545b3975f8c10c350249693b4553f774db6247fa65cc0d43cb9"} Nov 03 17:27:41.050804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:41.050765 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" event={"ID":"220de158-6794-4079-ac22-7969ed4a4699","Type":"ContainerStarted","Data":"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984"} Nov 03 17:27:41.051237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:41.050866 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:41.172782 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:41.172734 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" podStartSLOduration=2.295201395 podStartE2EDuration="3.17272112s" totalImagesPullingTime="877.519725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:41.171834998 +0000 UTC m=+603.626524023" watchObservedRunningTime="2026-11-03 17:27:41.17272112 +0000 UTC m=+603.627410145" Nov 03 17:27:41.313926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:41.313149 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:27:47.059915 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:47.059888 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:27:48.605501 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:27:48.605463 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:27:53.597106 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.597057 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:27:53.599830 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.599810 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632671 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632706 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632730 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632746 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632912 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.632977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.632958 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wmr9\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.633087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.633050 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.633132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.633112 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.633175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.633147 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.656717 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.656690 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2355f4-openshift-default-dockercfg-mdxdx\"" Nov 03 17:27:53.709172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.709143 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:27:53.734563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734527 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734579 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734607 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734641 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734666 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734708 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734974 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734744 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wmr9\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734974 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734813 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734974 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734847 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.734974 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.734923 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.735229 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.735099 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.735366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.735339 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.735484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.735388 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.735484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.735472 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.737456 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.737436 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.737624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.737523 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.810108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.810061 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wmr9\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.821844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.821816 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:53.912034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:53.911998 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:54.330942 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:27:54.330916 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3daef39_fd8a_403b_b772_16266a592579.slice/crio-3a22750dc27c10c97fc0744ab5fba9ec33db0adece66b380f3ff19ee8d5c0593 WatchSource:0}: Error finding container 3a22750dc27c10c97fc0744ab5fba9ec33db0adece66b380f3ff19ee8d5c0593: Status 404 returned error can't find the container with id 3a22750dc27c10c97fc0744ab5fba9ec33db0adece66b380f3ff19ee8d5c0593 Nov 03 17:27:54.331791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:54.331766 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:27:54.333491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:54.333458 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:27:54.333599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:54.333533 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:27:54.333599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:54.333577 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:27:55.130504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.130461 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" event={"ID":"c3daef39-fd8a-403b-b772-16266a592579","Type":"ContainerStarted","Data":"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957"} Nov 03 17:27:55.130929 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.130522 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" event={"ID":"c3daef39-fd8a-403b-b772-16266a592579","Type":"ContainerStarted","Data":"3a22750dc27c10c97fc0744ab5fba9ec33db0adece66b380f3ff19ee8d5c0593"} Nov 03 17:27:55.254029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.253983 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podStartSLOduration=2.253971115 podStartE2EDuration="2.253971115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:55.253576151 +0000 UTC m=+617.708265175" watchObservedRunningTime="2026-11-03 17:27:55.253971115 +0000 UTC m=+617.708660140" Nov 03 17:27:55.918512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.915414 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:27:55.923090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.922172 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 17:27:55.923090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:55.922230 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 17:27:56.912502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:56.912469 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 17:27:56.913124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:56.913058 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 17:27:57.084600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:57.084568 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:27:57.916457 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:57.916421 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 17:27:57.917096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:57.917044 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 17:27:58.912492 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:58.912450 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 17:27:58.912786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:58.912756 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 17:27:59.913625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:59.913582 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Nov 03 17:27:59.914269 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:27:59.914233 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Nov 03 17:28:00.923518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:00.923487 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:28:00.925059 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:00.925036 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:28:00.928690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:00.928672 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:28:10.314332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:10.314300 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:28:14.326749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.326712 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:28:14.331385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.331360 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.403293 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.403026 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:28:14.467255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467212 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467311 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5s9s\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467367 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467412 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467452 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467565 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467646 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467678 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.467753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.467741 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569096 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5s9s\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569165 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569206 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569242 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569274 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569303 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569326 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569514 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569554 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569857 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.569982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.569908 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.570210 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.570189 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.570468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.570392 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.570568 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.570546 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.571910 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.571888 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.572444 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.572425 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.632538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.632508 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.632672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.632652 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5s9s\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\") pod \"maas-default-gateway-openshift-default-58c6585447-swxpt\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:14.649085 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:14.649055 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:15.068207 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:28:15.068174 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17616578_c382_488d_8b91_adfe63a684b5.slice/crio-1b080a2b69a975292d5ba06db71fbf8b57ce2e2bd8a267e755b4c1d95e4ea5bd WatchSource:0}: Error finding container 1b080a2b69a975292d5ba06db71fbf8b57ce2e2bd8a267e755b4c1d95e4ea5bd: Status 404 returned error can't find the container with id 1b080a2b69a975292d5ba06db71fbf8b57ce2e2bd8a267e755b4c1d95e4ea5bd Nov 03 17:28:15.070615 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.070583 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:15.070703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.070648 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:15.070703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.070653 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:28:15.070703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.070677 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:15.258703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.257722 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" event={"ID":"17616578-c382-488d-8b91-adfe63a684b5","Type":"ContainerStarted","Data":"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e"} Nov 03 17:28:15.258703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.257831 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" event={"ID":"17616578-c382-488d-8b91-adfe63a684b5","Type":"ContainerStarted","Data":"1b080a2b69a975292d5ba06db71fbf8b57ce2e2bd8a267e755b4c1d95e4ea5bd"} Nov 03 17:28:15.383620 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.383573 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podStartSLOduration=1.3835599840000001 podStartE2EDuration="1.383559984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:15.382646952 +0000 UTC m=+637.837335976" watchObservedRunningTime="2026-11-03 17:28:15.383559984 +0000 UTC m=+637.838249008" Nov 03 17:28:15.651907 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:15.651800 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:16.650778 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:16.650744 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:16.651221 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:16.650799 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:17.649864 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:17.649831 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:17.650037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:17.649886 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:18.632323 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:28:18.632284 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:28:18.649833 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:18.649800 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:18.650024 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:18.649847 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:19.649582 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:19.649543 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:19.649949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:19.649609 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:20.649429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:20.649394 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:20.649601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:20.649460 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:21.650111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:21.650055 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:21.650508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:21.650156 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:22.649807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.649772 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 03 17:28:22.649982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.649823 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 03 17:28:22.962819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.962729 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:28:22.965851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.965832 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974044 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974012 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2jv\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974117 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974159 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974240 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974295 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974322 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974420 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974470 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:22.974615 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:22.974536 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.020440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.020409 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-af9e3c-openshift-default-dockercfg-pw9pq\"" Nov 03 17:28:23.072419 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.072378 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:28:23.075186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075162 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075317 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075199 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075317 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075221 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075317 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075249 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2jv\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075498 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075549 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075599 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075632 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075659 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075672 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.075832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075736 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.076008 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.075950 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.076087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.076007 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.076606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.076581 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.078277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.078249 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.078277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.078264 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.151103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.139255 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.151103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.139346 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2jv\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.278687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.278596 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:23.703195 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:28:23.703162 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea7afa7_5ff2_4cd7_8fbd_1f98b2e6049b.slice/crio-dd67b8f42d7c09e6b68b75844b943f596db2e1aa2aaca04c9a643f6a735d92e5 WatchSource:0}: Error finding container dd67b8f42d7c09e6b68b75844b943f596db2e1aa2aaca04c9a643f6a735d92e5: Status 404 returned error can't find the container with id dd67b8f42d7c09e6b68b75844b943f596db2e1aa2aaca04c9a643f6a735d92e5 Nov 03 17:28:23.704329 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.704310 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:28:23.706251 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.706211 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:23.706340 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.706318 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:23.706411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:23.706382 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:28:24.305957 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.305912 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerStarted","Data":"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce"} Nov 03 17:28:24.306506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.305978 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerStarted","Data":"dd67b8f42d7c09e6b68b75844b943f596db2e1aa2aaca04c9a643f6a735d92e5"} Nov 03 17:28:24.430392 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.430339 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podStartSLOduration=2.430326398 podStartE2EDuration="2.430326398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:24.429025555 +0000 UTC m=+646.883714579" watchObservedRunningTime="2026-11-03 17:28:24.430326398 +0000 UTC m=+646.885015423" Nov 03 17:28:24.650215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.650171 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-swxpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:28:24.650547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.650266 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:28:24.685161 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.685122 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:24.685760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.685730 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:24.686511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:24.686488 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:28:25.279833 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:25.279800 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:25.285547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:25.285518 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:25.310763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:25.310737 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:25.311616 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:25.311599 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:28:26.461112 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.461077 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:28:26.463618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.463602 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509258 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509325 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509422 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcs7\" (UniqueName: \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509520 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509484 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509521 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.509554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.509542 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.531131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.531096 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"default-dockercfg-r6smg\"" Nov 03 17:28:26.582460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.582429 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"simulator-w3-2355f4-kserve-self-signed-certs\"" Nov 03 17:28:26.582633 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.582511 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"openshift-service-ca.crt\"" Nov 03 17:28:26.584043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.584022 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:28:26.608398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.608369 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"kube-root-ca.crt\"" Nov 03 17:28:26.610471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610432 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.610632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610553 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.610632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610594 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcs7\" (UniqueName: \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.610754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610651 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.610754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610685 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.610754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.610712 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.611165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.611139 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.611812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.611789 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.612237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.612216 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.614337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.614317 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.620719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.620695 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.675751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.675723 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcs7\" (UniqueName: \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\") pod \"simulator-w3-2355f4-kserve-76bf65b475-w2rf6\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:26.789209 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:26.789115 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:27.167730 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:28:27.167702 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c8d9cc2_5208_424b_8008_ee4e84c9821c.slice/crio-272d5be4436b21ef544ca950d83f9c1b0e4955037f8db072976ee952509b710f WatchSource:0}: Error finding container 272d5be4436b21ef544ca950d83f9c1b0e4955037f8db072976ee952509b710f: Status 404 returned error can't find the container with id 272d5be4436b21ef544ca950d83f9c1b0e4955037f8db072976ee952509b710f Nov 03 17:28:27.169813 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:27.169798 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:28:27.169898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:27.169879 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:28:27.322437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:27.322403 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" event={"ID":"3c8d9cc2-5208-424b-8008-ee4e84c9821c","Type":"ContainerStarted","Data":"272d5be4436b21ef544ca950d83f9c1b0e4955037f8db072976ee952509b710f"} Nov 03 17:28:29.333749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:29.333712 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" event={"ID":"3c8d9cc2-5208-424b-8008-ee4e84c9821c","Type":"ContainerStarted","Data":"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb"} Nov 03 17:28:29.458891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:29.458826 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podStartSLOduration=1.871113875 podStartE2EDuration="3.458813241s" totalImagesPullingTime="1.587699366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:29.457318465 +0000 UTC m=+651.912007491" watchObservedRunningTime="2026-11-03 17:28:29.458813241 +0000 UTC m=+651.913502267" Nov 03 17:28:29.789876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:29.789834 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:30.410687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:30.410654 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:28:36.789931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:36.789897 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:36.802248 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:36.802226 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:37.382419 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:37.382393 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:28:43.770917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.770882 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:28:43.777308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.777291 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.829515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.829482 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:28:43.832004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.831978 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"openshift-service-ca.crt\"" Nov 03 17:28:43.832004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.831976 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"kube-root-ca.crt\"" Nov 03 17:28:43.879311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.879284 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"premium-w0-687fb3-kserve-self-signed-certs\"" Nov 03 17:28:43.879495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.879339 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"default-dockercfg-l8q7m\"" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924446 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8m4k\" (UniqueName: \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924535 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924618 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924675 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924702 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:43.927796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:43.924804 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025593 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025627 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025674 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025961 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025789 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8m4k\" (UniqueName: \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025961 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025838 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.025961 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.025890 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.026187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.026156 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.026257 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.026206 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.026365 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.026346 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.028283 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.028255 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.028676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.028656 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.089307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.089273 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8m4k\" (UniqueName: \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\") pod \"premium-w0-687fb3-kserve-579c56df4b-vdjfv\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.089500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.089481 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:44.513265 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:28:44.513223 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3a02bf5_bb41_40a7_92fb_68acb2d5cca8.slice/crio-07c9915680816b6c32633bf826056caaaf075371b89c5117c79f4e74627b34e1 WatchSource:0}: Error finding container 07c9915680816b6c32633bf826056caaaf075371b89c5117c79f4e74627b34e1: Status 404 returned error can't find the container with id 07c9915680816b6c32633bf826056caaaf075371b89c5117c79f4e74627b34e1 Nov 03 17:28:44.515422 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:44.515315 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:28:45.412777 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:45.412729 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" event={"ID":"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8","Type":"ContainerStarted","Data":"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4"} Nov 03 17:28:45.413284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:45.412786 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" event={"ID":"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8","Type":"ContainerStarted","Data":"07c9915680816b6c32633bf826056caaaf075371b89c5117c79f4e74627b34e1"} Nov 03 17:28:45.533950 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:45.533888 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podStartSLOduration=2.533870961 podStartE2EDuration="2.533870961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:45.533238971 +0000 UTC m=+667.987927996" watchObservedRunningTime="2026-11-03 17:28:45.533870961 +0000 UTC m=+667.988559987" Nov 03 17:28:46.090050 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:46.090009 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:48.606377 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:28:48.606343 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:28:49.392089 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:49.392038 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:28:54.090683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:54.090644 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:54.103204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:54.103179 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:28:54.468781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:28:54.468753 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:29:07.518092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.517997 4885 generic.go:356] "Generic (PLEG): container finished" podID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerID="c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce" exitCode=0 Nov 03 17:29:07.518594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.518087 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerDied","Data":"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce"} Nov 03 17:29:07.518594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.518545 4885 scope.go:122] "RemoveContainer" containerID="c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce" Nov 03 17:29:07.519773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.519743 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:07.519890 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.519810 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:07.519890 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.519849 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:07.749589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:07.749554 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:29:08.432971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.432934 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:29:08.435521 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.435503 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487280 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487326 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487388 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4n9\" (UniqueName: \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487422 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487466 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.487713 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.487691 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.496566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.496545 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"premium-w3-2355f4-kserve-self-signed-certs\"" Nov 03 17:29:08.540105 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.540055 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerStarted","Data":"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0"} Nov 03 17:29:08.548688 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.548662 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:29:08.588978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.588948 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589262 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589226 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589294 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589327 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4n9\" (UniqueName: \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589390 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589454 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.589568 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589540 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.590031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.589990 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.590171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.590057 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.592641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.592617 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.594336 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.594314 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.673107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.669701 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4n9\" (UniqueName: \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\") pod \"premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:08.745771 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:08.745682 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:09.140120 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:29:09.140084 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0926d730_d2c7_4feb_8c4d_eb1fac4943ac.slice/crio-6e9797c480304fd6ea7813c8eb9db7e9598b59c55d1cce8661d56fb07eae6fee WatchSource:0}: Error finding container 6e9797c480304fd6ea7813c8eb9db7e9598b59c55d1cce8661d56fb07eae6fee: Status 404 returned error can't find the container with id 6e9797c480304fd6ea7813c8eb9db7e9598b59c55d1cce8661d56fb07eae6fee Nov 03 17:29:09.140945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.140919 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:29:09.279407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.279377 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:29:09.280221 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.280191 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 17:29:09.280420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.280237 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 17:29:09.523141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.516732 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:29:09.527098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.527046 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.546590 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.546550 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" event={"ID":"0926d730-d2c7-4feb-8c4d-eb1fac4943ac","Type":"ContainerStarted","Data":"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2"} Nov 03 17:29:09.546941 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.546600 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" event={"ID":"0926d730-d2c7-4feb-8c4d-eb1fac4943ac","Type":"ContainerStarted","Data":"6e9797c480304fd6ea7813c8eb9db7e9598b59c55d1cce8661d56fb07eae6fee"} Nov 03 17:29:09.583053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.583000 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-687fb3-openshift-default-dockercfg-r5dqj\"" Nov 03 17:29:09.600625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600589 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.600792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600644 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thhg\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.600792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600722 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.600792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600761 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.600956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600794 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.601011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.600948 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.601101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.601050 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.601168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.601129 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.608090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.601542 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.609641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.609621 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:29:09.702992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.702933 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.704060 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.703272 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thhg\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.704315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.704283 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.704547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.704529 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.704765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.704748 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.704984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.704946 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.706614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.706594 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.706776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.706758 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.706934 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.706918 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.707569 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.707536 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.707714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.706059 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.709661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.708377 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.709661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.709611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.710518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.710499 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.728138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.728024 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.728848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.728808 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.746367 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.746340 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:09.766978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.766947 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thhg\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.775379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.775295 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:09.840269 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:09.840231 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:10.279835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.279795 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 17:29:10.280007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.279854 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 17:29:10.314266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.314177 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:29:10.370203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.370149 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podStartSLOduration=2.370135229 podStartE2EDuration="2.370135229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:09.781750723 +0000 UTC m=+692.236439750" watchObservedRunningTime="2026-11-03 17:29:10.370135229 +0000 UTC m=+692.824824254" Nov 03 17:29:10.371244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.371224 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:29:10.388255 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:29:10.388229 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f3ccaa_f412_464d_890f_8524efbf560d.slice/crio-7e1710b4980c811b925dcec5c57e3e704b829b829e75f8d25b5b6c4095b89f13 WatchSource:0}: Error finding container 7e1710b4980c811b925dcec5c57e3e704b829b829e75f8d25b5b6c4095b89f13: Status 404 returned error can't find the container with id 7e1710b4980c811b925dcec5c57e3e704b829b829e75f8d25b5b6c4095b89f13 Nov 03 17:29:10.390778 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.390749 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:10.390858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.390813 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:10.390858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.390841 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:29:10.556123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.556007 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" event={"ID":"66f3ccaa-f412-464d-890f-8524efbf560d","Type":"ContainerStarted","Data":"b880f99f8c0339e00194a454eaac7886f7085ec582027d2060585d4f9b047a54"} Nov 03 17:29:10.556123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.556088 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" event={"ID":"66f3ccaa-f412-464d-890f-8524efbf560d","Type":"ContainerStarted","Data":"7e1710b4980c811b925dcec5c57e3e704b829b829e75f8d25b5b6c4095b89f13"} Nov 03 17:29:10.682471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.682416 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podStartSLOduration=1.6824008080000001 podStartE2EDuration="1.682400808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:10.681157259 +0000 UTC m=+693.135846297" watchObservedRunningTime="2026-11-03 17:29:10.682400808 +0000 UTC m=+693.137089833" Nov 03 17:29:10.840624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:10.840509 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:11.279229 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:11.279198 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 17:29:11.279401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:11.279247 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 17:29:11.843135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:11.843099 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:11.843731 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:11.843684 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:12.279154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:12.279119 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 17:29:12.279334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:12.279182 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 17:29:12.844917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:12.844465 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:12.844917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:12.844530 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:13.279251 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:13.279214 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 17:29:13.279497 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:13.279477 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 17:29:13.841570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:13.841533 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:13.841876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:13.841828 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:14.283694 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:14.283663 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:29:14.284618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:14.284596 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:29:14.841588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:14.841553 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:14.841762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:14.841605 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:15.841162 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:15.841125 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:15.841548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:15.841181 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:16.841415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:16.841380 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 17:29:16.841816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:16.841466 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 17:29:18.352571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:18.352536 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:18.353637 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:18.353611 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:18.354643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:18.354617 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:29:18.605656 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:29:18.605580 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:29:18.746002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:18.745975 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:18.758501 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:18.758473 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:19.615660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:19.615632 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:29:27.517878 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:27.517850 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:29:45.324791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:29:45.324756 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:29:48.607767 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:29:48.607726 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:30:00.197806 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.197715 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:00.201753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.201728 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.212433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.212387 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:00.216368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.216348 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.266533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.266500 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:30:00.292555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.292523 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:00.311491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.311460 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:00.327791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.327755 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.327942 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.327809 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.327982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.327950 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbbt\" (UniqueName: \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.328017 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.328003 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8mhm\" (UniqueName: \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.450273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.450179 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.459095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.450494 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.459477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.459449 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbbt\" (UniqueName: \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.459642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.459612 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.459720 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.458921 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.460162 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.460140 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8mhm\" (UniqueName: \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.528164 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.528136 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8mhm\" (UniqueName: \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.528756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.528725 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbbt\" (UniqueName: \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\") pod \"maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.532510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.532427 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:00.817624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.817549 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:00.933405 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:30:00.933377 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9ae5fa_1037_4052_a854_6e82524704cc.slice/crio-12b1c6bc5ef9d8a0666a636de81b9549bf47c13d686c30fd663058a0f9e22766 WatchSource:0}: Error finding container 12b1c6bc5ef9d8a0666a636de81b9549bf47c13d686c30fd663058a0f9e22766: Status 404 returned error can't find the container with id 12b1c6bc5ef9d8a0666a636de81b9549bf47c13d686c30fd663058a0f9e22766 Nov 03 17:30:00.934025 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:00.933992 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:01.187806 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:30:01.187772 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15363634_0eb7_43cb_8676_d61c3e2414e5.slice/crio-e311c4f51a18dddcf8610667847dd40257b2b7f793cbc75f416b895411f4aa33 WatchSource:0}: Error finding container e311c4f51a18dddcf8610667847dd40257b2b7f793cbc75f416b895411f4aa33: Status 404 returned error can't find the container with id e311c4f51a18dddcf8610667847dd40257b2b7f793cbc75f416b895411f4aa33 Nov 03 17:30:01.189458 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:01.189366 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:01.868139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:01.868102 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerStarted","Data":"12b1c6bc5ef9d8a0666a636de81b9549bf47c13d686c30fd663058a0f9e22766"} Nov 03 17:30:01.869410 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:01.869385 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerStarted","Data":"e311c4f51a18dddcf8610667847dd40257b2b7f793cbc75f416b895411f4aa33"} Nov 03 17:30:06.896038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:06.896005 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerStarted","Data":"c4ea1cb469a1eb3495add1cbb2612a132ed9256a6ed53443abdf0134901d33d3"} Nov 03 17:30:07.018171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:07.018122 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" podStartSLOduration=1.434405194 podStartE2EDuration="7.018110118s" totalImagesPullingTime="5.583704924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:07.01747827 +0000 UTC m=+749.472167296" watchObservedRunningTime="2026-11-03 17:30:07.018110118 +0000 UTC m=+749.472799143" Nov 03 17:30:07.901035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:07.900994 4885 generic.go:356] "Generic (PLEG): container finished" podID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerID="c4ea1cb469a1eb3495add1cbb2612a132ed9256a6ed53443abdf0134901d33d3" exitCode=60 Nov 03 17:30:07.901524 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:07.901031 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerDied","Data":"c4ea1cb469a1eb3495add1cbb2612a132ed9256a6ed53443abdf0134901d33d3"} Nov 03 17:30:07.901524 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:07.901383 4885 scope.go:122] "RemoveContainer" containerID="c4ea1cb469a1eb3495add1cbb2612a132ed9256a6ed53443abdf0134901d33d3" Nov 03 17:30:08.908413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:08.908384 4885 generic.go:356] "Generic (PLEG): container finished" podID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerID="31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe" exitCode=60 Nov 03 17:30:08.908861 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:08.908429 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerDied","Data":"31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe"} Nov 03 17:30:08.908861 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:08.908476 4885 scope.go:122] "RemoveContainer" containerID="c4ea1cb469a1eb3495add1cbb2612a132ed9256a6ed53443abdf0134901d33d3" Nov 03 17:30:08.908861 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:08.908745 4885 scope.go:122] "RemoveContainer" containerID="31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe" Nov 03 17:30:08.909038 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:08.909018 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n_odh-ai-gateway-infra(0b9ae5fa-1037-4052-a854-6e82524704cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" Nov 03 17:30:09.914505 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:09.914470 4885 scope.go:122] "RemoveContainer" containerID="31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe" Nov 03 17:30:09.914934 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:09.914718 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n_odh-ai-gateway-infra(0b9ae5fa-1037-4052-a854-6e82524704cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" Nov 03 17:30:09.915778 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:09.915752 4885 generic.go:356] "Generic (PLEG): container finished" podID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerID="9148070e17803cb220fd359d8a2f90cb2ee45c30001214641a969417d5b6ffdf" exitCode=60 Nov 03 17:30:09.915875 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:09.915802 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerDied","Data":"9148070e17803cb220fd359d8a2f90cb2ee45c30001214641a969417d5b6ffdf"} Nov 03 17:30:09.916025 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:09.916014 4885 scope.go:122] "RemoveContainer" containerID="9148070e17803cb220fd359d8a2f90cb2ee45c30001214641a969417d5b6ffdf" Nov 03 17:30:10.405785 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.405750 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr"] Nov 03 17:30:10.409831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.409809 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.465290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.464183 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr"] Nov 03 17:30:10.465290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.464974 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:30:10.476751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.476723 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b5r\" (UniqueName: \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.476879 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.476771 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.476879 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.476863 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.577630 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.577596 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b5r\" (UniqueName: \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.577829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.577645 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.577829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.577671 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.578527 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.578505 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.580234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.580216 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.644557 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.644522 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b5r\" (UniqueName: \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\") pod \"maas-api-revoke-sub-f0e0c045-mvzqr\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.723892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.723808 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:10.922387 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.922355 4885 generic.go:356] "Generic (PLEG): container finished" podID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerID="316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f" exitCode=60 Nov 03 17:30:10.922781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.922478 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerDied","Data":"316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f"} Nov 03 17:30:10.922781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.922543 4885 scope.go:122] "RemoveContainer" containerID="9148070e17803cb220fd359d8a2f90cb2ee45c30001214641a969417d5b6ffdf" Nov 03 17:30:10.922781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:10.922696 4885 scope.go:122] "RemoveContainer" containerID="316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f" Nov 03 17:30:10.922985 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:10.922965 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8_odh-ai-gateway-infra(15363634-0eb7-43cb-8676-d61c3e2414e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" Nov 03 17:30:11.095341 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:30:11.095305 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c61a0c_14bb_4e5c_b0e8_d5a58ace2d00.slice/crio-815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9 WatchSource:0}: Error finding container 815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9: Status 404 returned error can't find the container with id 815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9 Nov 03 17:30:11.097647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:11.097625 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr"] Nov 03 17:30:11.929608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:11.929568 4885 scope.go:122] "RemoveContainer" containerID="316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f" Nov 03 17:30:11.930153 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:11.929822 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8_odh-ai-gateway-infra(15363634-0eb7-43cb-8676-d61c3e2414e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" Nov 03 17:30:11.930971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:11.930942 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" event={"ID":"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00","Type":"ContainerStarted","Data":"6064b2f9a3eacc95e522a8c0591192da0300adeb08a41986956a4d2ff5d7b539"} Nov 03 17:30:11.931135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:11.930996 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" event={"ID":"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00","Type":"ContainerStarted","Data":"815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9"} Nov 03 17:30:12.168719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:12.168669 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" podStartSLOduration=2.168654047 podStartE2EDuration="2.168654047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:12.166943208 +0000 UTC m=+754.621632238" watchObservedRunningTime="2026-11-03 17:30:12.168654047 +0000 UTC m=+754.623343072" Nov 03 17:30:12.758988 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:12.758952 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:30:12.937595 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:12.937557 4885 generic.go:356] "Generic (PLEG): container finished" podID="34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" containerID="6064b2f9a3eacc95e522a8c0591192da0300adeb08a41986956a4d2ff5d7b539" exitCode=0 Nov 03 17:30:12.937999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:12.937643 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" event={"ID":"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00","Type":"ContainerDied","Data":"6064b2f9a3eacc95e522a8c0591192da0300adeb08a41986956a4d2ff5d7b539"} Nov 03 17:30:14.093764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.093740 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:14.123462 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.123437 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\") pod \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " Nov 03 17:30:14.123621 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.123537 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\" (UniqueName: \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\") pod \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " Nov 03 17:30:14.123621 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.123569 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\") pod \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\" (UID: \"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00\") " Nov 03 17:30:14.124159 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.124119 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca" pod "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" (UID: "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:14.126472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.126442 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca" pod "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" (UID: "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:14.126472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.126456 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r" pod "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" (UID: "34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00"). InnerVolumeSpecName "kube-api-access-v2b5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:14.226158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.226012 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2b5r\" (UniqueName: \"kubernetes.io/projected/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-kube-api-access-v2b5r\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:14.226158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.226046 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:14.226158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.226078 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:14.949592 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.949563 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" Nov 03 17:30:14.949592 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.949583 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr" event={"ID":"34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00","Type":"ContainerDied","Data":"815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9"} Nov 03 17:30:14.949842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:14.949626 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815cc3d9aef274e2e15b90df24d05dff8cce28a1d3668e42ba93ba11f757d5c9" Nov 03 17:30:18.606441 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:18.606393 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:30:20.499020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.498984 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:30:20.499578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.499559 4885 cpu_manager.go:414] "containerMap: removing container" podUID="34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" containerName="revoke-keys" Nov 03 17:30:20.499661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.499580 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" containerName="revoke-keys" Nov 03 17:30:20.499732 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.499719 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" containerName="revoke-keys" Nov 03 17:30:20.502807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.502785 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599495 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599569 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599622 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599708 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdrhk\" (UniqueName: \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599738 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.599820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.599816 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.610992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.610958 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2355f4\"/\"e2e-conflict-a-0ff8cfd1-kserve-self-signed-certs\"" Nov 03 17:30:20.613290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.613264 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:30:20.700685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700651 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.700860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700699 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.700860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700727 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.700860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700794 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdrhk\" (UniqueName: \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.700860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700820 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.701108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.700880 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.701210 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.701184 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.701360 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.701246 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.701507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.701483 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.703433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.703403 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.703723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.703703 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.773938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.773858 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdrhk\" (UniqueName: \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\") pod \"e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:20.816502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:20.816281 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:21.227161 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:30:21.227132 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82e0178e_339d_4632_a606_431ac81f932c.slice/crio-e929e07a9218848a5a7d7cb2f21a62a624fee2695ec1ecf54321d678d283a808 WatchSource:0}: Error finding container e929e07a9218848a5a7d7cb2f21a62a624fee2695ec1ecf54321d678d283a808: Status 404 returned error can't find the container with id e929e07a9218848a5a7d7cb2f21a62a624fee2695ec1ecf54321d678d283a808 Nov 03 17:30:21.228228 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:21.228203 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:30:21.986898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:21.986860 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" event={"ID":"82e0178e-339d-4632-a606-431ac81f932c","Type":"ContainerStarted","Data":"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8"} Nov 03 17:30:21.986898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:21.986904 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" event={"ID":"82e0178e-339d-4632-a606-431ac81f932c","Type":"ContainerStarted","Data":"e929e07a9218848a5a7d7cb2f21a62a624fee2695ec1ecf54321d678d283a808"} Nov 03 17:30:22.111917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:22.111864 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" podStartSLOduration=2.111848946 podStartE2EDuration="2.111848946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:22.110206307 +0000 UTC m=+764.564895336" watchObservedRunningTime="2026-11-03 17:30:22.111848946 +0000 UTC m=+764.566537973" Nov 03 17:30:22.818016 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:22.817981 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:23.656197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:23.656171 4885 scope.go:122] "RemoveContainer" containerID="31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe" Nov 03 17:30:23.997762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:23.997727 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerStarted","Data":"c3da1937253f69ce73391a56a7007e67f1daad2d6d446ef886e02f3655ef9d87"} Nov 03 17:30:24.656081 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:24.656036 4885 scope.go:122] "RemoveContainer" containerID="316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f" Nov 03 17:30:25.004883 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.004794 4885 generic.go:356] "Generic (PLEG): container finished" podID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerID="c3da1937253f69ce73391a56a7007e67f1daad2d6d446ef886e02f3655ef9d87" exitCode=60 Nov 03 17:30:25.004883 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.004867 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerDied","Data":"c3da1937253f69ce73391a56a7007e67f1daad2d6d446ef886e02f3655ef9d87"} Nov 03 17:30:25.005687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.004928 4885 scope.go:122] "RemoveContainer" containerID="31168d66290215d9bd653441868f469ff601827890d9dcc125b59bff9d832dbe" Nov 03 17:30:25.005687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.005273 4885 scope.go:122] "RemoveContainer" containerID="c3da1937253f69ce73391a56a7007e67f1daad2d6d446ef886e02f3655ef9d87" Nov 03 17:30:25.005687 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:25.005536 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n_odh-ai-gateway-infra(0b9ae5fa-1037-4052-a854-6e82524704cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" Nov 03 17:30:25.006931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.006913 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerStarted","Data":"816f41c49ad9c6a55c85593a99465ea50aeeb4855d31ef650081cca1bddf50f3"} Nov 03 17:30:25.136900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.136863 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:25.244050 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:25.243989 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" podStartSLOduration=17.737908455 podStartE2EDuration="25.24396686s" totalImagesPullingTime="7.506058405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:25.24335224 +0000 UTC m=+767.698041264" watchObservedRunningTime="2026-11-03 17:30:25.24396686 +0000 UTC m=+767.698655885" Nov 03 17:30:26.015511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.015477 4885 generic.go:356] "Generic (PLEG): container finished" podID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerID="816f41c49ad9c6a55c85593a99465ea50aeeb4855d31ef650081cca1bddf50f3" exitCode=60 Nov 03 17:30:26.015835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.015547 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerDied","Data":"816f41c49ad9c6a55c85593a99465ea50aeeb4855d31ef650081cca1bddf50f3"} Nov 03 17:30:26.015835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.015609 4885 scope.go:122] "RemoveContainer" containerID="316ab1422cfa5511b075040875e3b8834f29eddd7be3e3d9fbf621b5dbaca13f" Nov 03 17:30:26.015994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.015974 4885 scope.go:122] "RemoveContainer" containerID="816f41c49ad9c6a55c85593a99465ea50aeeb4855d31ef650081cca1bddf50f3" Nov 03 17:30:26.016372 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:26.016346 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8_odh-ai-gateway-infra(15363634-0eb7-43cb-8676-d61c3e2414e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" Nov 03 17:30:26.159932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.159909 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:26.258854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.258823 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:26.264333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.264245 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\" (UniqueName: \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\") pod \"0b9ae5fa-1037-4052-a854-6e82524704cc\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " Nov 03 17:30:26.264495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.264393 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\") pod \"0b9ae5fa-1037-4052-a854-6e82524704cc\" (UID: \"0b9ae5fa-1037-4052-a854-6e82524704cc\") " Nov 03 17:30:26.264809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.264778 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca" pod "0b9ae5fa-1037-4052-a854-6e82524704cc" (UID: "0b9ae5fa-1037-4052-a854-6e82524704cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:26.267081 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.267044 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm" pod "0b9ae5fa-1037-4052-a854-6e82524704cc" (UID: "0b9ae5fa-1037-4052-a854-6e82524704cc"). InnerVolumeSpecName "kube-api-access-n8mhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:26.365558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.365527 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9ae5fa-1037-4052-a854-6e82524704cc-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:26.365558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:26.365554 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8mhm\" (UniqueName: \"kubernetes.io/projected/0b9ae5fa-1037-4052-a854-6e82524704cc-kube-api-access-n8mhm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:27.030594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.030561 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" Nov 03 17:30:27.031053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.031024 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n" event={"ID":"0b9ae5fa-1037-4052-a854-6e82524704cc","Type":"ContainerDied","Data":"12b1c6bc5ef9d8a0666a636de81b9549bf47c13d686c30fd663058a0f9e22766"} Nov 03 17:30:27.031305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.031280 4885 scope.go:122] "RemoveContainer" containerID="c3da1937253f69ce73391a56a7007e67f1daad2d6d446ef886e02f3655ef9d87" Nov 03 17:30:27.187662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.187639 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:27.273330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.273299 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\" (UniqueName: \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\") pod \"15363634-0eb7-43cb-8676-d61c3e2414e5\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " Nov 03 17:30:27.273497 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.273336 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\") pod \"15363634-0eb7-43cb-8676-d61c3e2414e5\" (UID: \"15363634-0eb7-43cb-8676-d61c3e2414e5\") " Nov 03 17:30:27.273756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.273737 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca" pod "15363634-0eb7-43cb-8676-d61c3e2414e5" (UID: "15363634-0eb7-43cb-8676-d61c3e2414e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:27.275498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.275469 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt" pod "15363634-0eb7-43cb-8676-d61c3e2414e5" (UID: "15363634-0eb7-43cb-8676-d61c3e2414e5"). InnerVolumeSpecName "kube-api-access-vdbbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:27.341532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.341499 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:27.346273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.346251 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895450-szg4n"] Nov 03 17:30:27.375062 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.375035 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbbt\" (UniqueName: \"kubernetes.io/projected/15363634-0eb7-43cb-8676-d61c3e2414e5-kube-api-access-vdbbt\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:27.375062 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.375058 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15363634-0eb7-43cb-8676-d61c3e2414e5-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:30:27.665090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:27.665027 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" path="/var/lib/kubelet/pods/0b9ae5fa-1037-4052-a854-6e82524704cc/volumes" Nov 03 17:30:28.039358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.039278 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" event={"ID":"15363634-0eb7-43cb-8676-d61c3e2414e5","Type":"ContainerDied","Data":"e311c4f51a18dddcf8610667847dd40257b2b7f793cbc75f416b895411f4aa33"} Nov 03 17:30:28.039358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.039296 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8" Nov 03 17:30:28.039358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.039336 4885 scope.go:122] "RemoveContainer" containerID="816f41c49ad9c6a55c85593a99465ea50aeeb4855d31ef650081cca1bddf50f3" Nov 03 17:30:28.208232 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.208197 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:28.214262 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.214240 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-8fc5ec-29895450-ld9v8"] Nov 03 17:30:28.251556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:28.251530 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:30:29.666131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:29.666097 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" path="/var/lib/kubelet/pods/15363634-0eb7-43cb-8676-d61c3e2414e5/volumes" Nov 03 17:30:30.818489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:30.818454 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:30.831530 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:30.831504 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:31.068461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:31.068432 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:30:48.604772 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:30:48.604732 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:30:54.056842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.056809 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057186 4885 cpu_manager.go:414] "containerMap: removing container" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057197 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057216 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057221 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057230 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057235 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057245 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057250 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057258 4885 cpu_manager.go:414] "containerMap: removing container" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057262 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057273 4885 cpu_manager.go:414] "containerMap: removing container" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057278 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057330 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.057669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057340 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057349 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057358 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b9ae5fa-1037-4052-a854-6e82524704cc" containerName="cleanup" Nov 03 17:30:54.057669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.057367 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:30:54.063416 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.063395 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.119418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.119389 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:30:54.165916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.165888 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.165916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.165918 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.166154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.166084 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.166154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.166140 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.166237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.166191 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.166237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.166211 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t5p\" (UniqueName: \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.168123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.168103 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"auth-test-model-af9e3c-kserve-self-signed-certs\"" Nov 03 17:30:54.168370 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.168357 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"default-dockercfg-bl29d\"" Nov 03 17:30:54.176750 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.176732 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"openshift-service-ca.crt\"" Nov 03 17:30:54.176839 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.176791 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-af9e3c\"/\"kube-root-ca.crt\"" Nov 03 17:30:54.266907 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.266872 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.266926 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.266946 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.266965 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t5p\" (UniqueName: \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.267018 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.267048 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267454 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.267426 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267633 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.267452 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.267744 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.267476 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.269438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.269418 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.269930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.269912 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.330856 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.330769 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t5p\" (UniqueName: \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\") pod \"auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.375390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.375358 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:30:54.763206 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:30:54.763177 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1025f336_45aa_4d1c_98c9_33127113b59e.slice/crio-9dffc10e404a349044e0bccce1b5d7ab3f45bff4a67436767d88ac341cd76f84 WatchSource:0}: Error finding container 9dffc10e404a349044e0bccce1b5d7ab3f45bff4a67436767d88ac341cd76f84: Status 404 returned error can't find the container with id 9dffc10e404a349044e0bccce1b5d7ab3f45bff4a67436767d88ac341cd76f84 Nov 03 17:30:54.765118 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:54.765099 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:30:55.143486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:55.143447 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:30:55.174812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:55.174779 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" event={"ID":"1025f336-45aa-4d1c-98c9-33127113b59e","Type":"ContainerStarted","Data":"5d4e51424f0cccfb1c763413502dc1962639151517700e01971b35883ad424e2"} Nov 03 17:30:55.175016 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:55.174820 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" event={"ID":"1025f336-45aa-4d1c-98c9-33127113b59e","Type":"ContainerStarted","Data":"9dffc10e404a349044e0bccce1b5d7ab3f45bff4a67436767d88ac341cd76f84"} Nov 03 17:30:55.298612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:55.298556 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" podStartSLOduration=1.298540454 podStartE2EDuration="1.298540454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:55.297037017 +0000 UTC m=+797.751726043" watchObservedRunningTime="2026-11-03 17:30:55.298540454 +0000 UTC m=+797.753229479" Nov 03 17:30:55.375759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:30:55.375731 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:31:04.375767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:04.375731 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:31:04.391801 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:04.391777 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:31:05.232866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:05.232838 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:31:18.606754 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:31:18.606717 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:31:20.925943 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:20.925909 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:31:25.591332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.591297 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:31:25.591799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.591784 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="15363634-0eb7-43cb-8676-d61c3e2414e5" containerName="cleanup" Nov 03 17:31:25.595217 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.595192 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.643098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.643031 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:31:25.646821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.646797 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.707501 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.707472 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:31:25.707952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.707929 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.708047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.707984 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.708047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.708019 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.708168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.708099 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.708168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.708140 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.708267 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.708165 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709360 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709339 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.709466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709368 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709386 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.709466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709407 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709602 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709530 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xrl\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709598 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709656 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.709714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709688 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.709829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709748 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709803 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.709997 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.709975 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ff6\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.710105 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.710045 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.764142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.764111 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830502 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830565 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830608 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xrl\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830640 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830665 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830692 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830728 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830758 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830808 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ff6\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830851 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830881 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830921 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830953 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.830982 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.831022 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.831050 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.831152 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.831182 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.837906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.836572 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.837906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.836825 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.837906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.837320 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.837906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.837828 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.838138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.838014 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.838257 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.838731 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.847557 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.847597 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.848234 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.852611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.857507 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.864335 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.870897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.864634 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.903238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.903207 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.903381 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.903362 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xrl\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.905892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.905870 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.911456 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.911431 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ff6\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:25.950638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.950599 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:25.961103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:25.961042 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:26.418451 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:31:26.418424 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977dfe00_1104_48ed_8df6_2f90673fbf92.slice/crio-c8180b466f5b7f541f62ef38af24bce8d612c687035231a6e801f99d119fdd07 WatchSource:0}: Error finding container c8180b466f5b7f541f62ef38af24bce8d612c687035231a6e801f99d119fdd07: Status 404 returned error can't find the container with id c8180b466f5b7f541f62ef38af24bce8d612c687035231a6e801f99d119fdd07 Nov 03 17:31:26.420648 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.420629 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:31:26.421349 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.421319 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:26.421430 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.421380 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:26.421430 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.421406 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:26.472291 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.472265 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:31:26.472989 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:31:26.472967 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657602cf_f855_45bc_89f4_2af0ec76e834.slice/crio-3bceba8cbcb018964a0bbc6a868e468007fb8cf275009c5d8bbafadd1be9b258 WatchSource:0}: Error finding container 3bceba8cbcb018964a0bbc6a868e468007fb8cf275009c5d8bbafadd1be9b258: Status 404 returned error can't find the container with id 3bceba8cbcb018964a0bbc6a868e468007fb8cf275009c5d8bbafadd1be9b258 Nov 03 17:31:26.475988 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.475950 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:26.476100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.476035 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:26.476171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:26.476110 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:27.355052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.355012 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" event={"ID":"657602cf-f855-45bc-89f4-2af0ec76e834","Type":"ContainerStarted","Data":"837c82cfa3733b6fed63e1f20bff6f9de625a457d12bb053b35e834c3c172279"} Nov 03 17:31:27.355580 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.355144 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" event={"ID":"657602cf-f855-45bc-89f4-2af0ec76e834","Type":"ContainerStarted","Data":"3bceba8cbcb018964a0bbc6a868e468007fb8cf275009c5d8bbafadd1be9b258"} Nov 03 17:31:27.356724 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.356697 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" event={"ID":"977dfe00-1104-48ed-8df6-2f90673fbf92","Type":"ContainerStarted","Data":"e551b0abdf015f7f7ebd4e3e85f0be3b8ba454262d2e86dfee63cf9d5d11afb3"} Nov 03 17:31:27.356842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.356735 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" event={"ID":"977dfe00-1104-48ed-8df6-2f90673fbf92","Type":"ContainerStarted","Data":"c8180b466f5b7f541f62ef38af24bce8d612c687035231a6e801f99d119fdd07"} Nov 03 17:31:27.561127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.538833 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podStartSLOduration=2.538811364 podStartE2EDuration="2.538811364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:27.536614138 +0000 UTC m=+829.991303166" watchObservedRunningTime="2026-11-03 17:31:27.538811364 +0000 UTC m=+829.993500390" Nov 03 17:31:27.953713 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.952388 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:27.957198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.954233 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:27.957198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.954297 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:27.961356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.961338 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:27.962754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.962733 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:27.962858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:27.962790 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:28.951500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:28.951460 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:28.952129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:28.952092 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:28.961567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:28.961536 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:28.961795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:28.961767 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:29.959129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:29.955513 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:29.959129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:29.955578 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:29.965612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:29.965582 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:29.965916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:29.965887 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:30.953578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:30.953540 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:30.953882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:30.953833 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:30.967618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:30.965417 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:30.967618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:30.965471 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:31.959101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:31.952567 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:31.959101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:31.952628 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:31.971093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:31.964510 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:31.971093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:31.964589 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:32.951170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:32.951121 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:31:32.951380 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:32.951197 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 03 17:31:32.961478 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:32.961443 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 17:31:32.961616 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:32.961502 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 17:31:33.863526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.863465 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podStartSLOduration=8.863447821 podStartE2EDuration="8.863447821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:27.668736158 +0000 UTC m=+830.123425184" watchObservedRunningTime="2026-11-03 17:31:33.863447821 +0000 UTC m=+836.318136846" Nov 03 17:31:33.865415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.865391 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8"] Nov 03 17:31:33.869544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.869498 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:33.904844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.904790 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:33.904844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.904845 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:33.905127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.904874 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcm8\" (UniqueName: \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:33.926405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.926372 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8"] Nov 03 17:31:33.978101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.974318 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:31:33.978101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:33.974444 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:31:34.013014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.011582 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.013014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.011631 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcm8\" (UniqueName: \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.013014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.011836 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.018898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.018871 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.021657 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.021526 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:34.022785 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.022682 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:34.023941 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.023922 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:31:34.024358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.024277 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.025109 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.025030 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:34.025726 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.025711 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:34.026783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.026766 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:31:34.075732 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.075698 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcm8\" (UniqueName: \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\") pod \"maas-api-revoke-sub-739e8d90-wbdh8\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.186621 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.186586 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:34.572127 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:31:34.569213 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f877f0_5ba2_4a8d_bb58_8286395b9dcb.slice/crio-e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8 WatchSource:0}: Error finding container e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8: Status 404 returned error can't find the container with id e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8 Nov 03 17:31:34.572127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:34.570582 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8"] Nov 03 17:31:35.399176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:35.399139 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" event={"ID":"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb","Type":"ContainerStarted","Data":"02827f4ffd2ac42226acd1d66a36acfd29c90a1a2e053744bee3b2cc6adc5ea1"} Nov 03 17:31:35.399577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:35.399201 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" event={"ID":"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb","Type":"ContainerStarted","Data":"e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8"} Nov 03 17:31:35.518409 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:35.518360 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" podStartSLOduration=2.518345117 podStartE2EDuration="2.518345117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:35.51799181 +0000 UTC m=+837.972680836" watchObservedRunningTime="2026-11-03 17:31:35.518345117 +0000 UTC m=+837.973034141" Nov 03 17:31:36.405692 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:36.405655 4885 generic.go:356] "Generic (PLEG): container finished" podID="a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" containerID="02827f4ffd2ac42226acd1d66a36acfd29c90a1a2e053744bee3b2cc6adc5ea1" exitCode=0 Nov 03 17:31:36.406125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:36.405749 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" event={"ID":"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb","Type":"ContainerDied","Data":"02827f4ffd2ac42226acd1d66a36acfd29c90a1a2e053744bee3b2cc6adc5ea1"} Nov 03 17:31:37.569389 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.569362 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:37.668884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.668817 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\" (UniqueName: \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\") pod \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " Nov 03 17:31:37.668884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.668871 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\") pod \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " Nov 03 17:31:37.669127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.668948 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\") pod \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\" (UID: \"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb\") " Nov 03 17:31:37.669292 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.669264 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca" pod "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" (UID: "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:37.670878 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.670851 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8" pod "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" (UID: "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb"). InnerVolumeSpecName "kube-api-access-cgcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:37.670977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.670957 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca" pod "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" (UID: "a2f877f0-5ba2-4a8d-bb58-8286395b9dcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:37.770366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.770336 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcm8\" (UniqueName: \"kubernetes.io/projected/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-kube-api-access-cgcm8\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:31:37.770366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.770362 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:31:37.770366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:37.770371 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:31:38.416301 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:38.416264 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" event={"ID":"a2f877f0-5ba2-4a8d-bb58-8286395b9dcb","Type":"ContainerDied","Data":"e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8"} Nov 03 17:31:38.416491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:38.416317 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8" Nov 03 17:31:38.416491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:38.416325 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d6a6c37e1dffc1b72c4f982fe2fdc90a013a743e23a6ae67173e5c871ea7c8" Nov 03 17:31:42.671966 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:42.671934 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:31:45.660132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.651918 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:31:45.660132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.657381 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" containerName="revoke-keys" Nov 03 17:31:45.660132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.657461 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" containerName="revoke-keys" Nov 03 17:31:45.660132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.657574 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" containerName="revoke-keys" Nov 03 17:31:45.670812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.670782 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.714247 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.714218 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:31:45.762990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.762951 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"default-dockercfg-78pkk\"" Nov 03 17:31:45.763197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.763025 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"auth-test-model-147834-kserve-self-signed-certs\"" Nov 03 17:31:45.776782 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.776750 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"kube-root-ca.crt\"" Nov 03 17:31:45.776782 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.776761 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"openshift-service-ca.crt\"" Nov 03 17:31:45.812138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.790344 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.812517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.812488 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmvc9\" (UniqueName: \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.812724 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.812695 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.812944 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.812924 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.813324 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.813304 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.813479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.813463 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915217 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915131 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915237 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915281 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915332 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915357 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmvc9\" (UniqueName: \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915391 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915608 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915735 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.915825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.915777 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.917892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.917862 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.918008 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.917986 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:45.978650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:45.978614 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmvc9\" (UniqueName: \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\") pod \"auth-test-model-147834-kserve-58fdcb9cdf-6lj28\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:46.015757 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:46.015506 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:46.386249 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:31:46.386206 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3239d83_1741_44dc_95eb_2932236c2a02.slice/crio-6a2fa05b313f54f7031b0cf9dc32e0080c39d164e79240b614a0d9b103b05657 WatchSource:0}: Error finding container 6a2fa05b313f54f7031b0cf9dc32e0080c39d164e79240b614a0d9b103b05657: Status 404 returned error can't find the container with id 6a2fa05b313f54f7031b0cf9dc32e0080c39d164e79240b614a0d9b103b05657 Nov 03 17:31:46.388909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:46.388881 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:31:46.467498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:46.467461 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" event={"ID":"c3239d83-1741-44dc-95eb-2932236c2a02","Type":"ContainerStarted","Data":"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba"} Nov 03 17:31:46.467644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:46.467518 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" event={"ID":"c3239d83-1741-44dc-95eb-2932236c2a02","Type":"ContainerStarted","Data":"6a2fa05b313f54f7031b0cf9dc32e0080c39d164e79240b614a0d9b103b05657"} Nov 03 17:31:46.587653 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:46.587603 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podStartSLOduration=1.587589043 podStartE2EDuration="1.587589043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:46.587179745 +0000 UTC m=+849.041868782" watchObservedRunningTime="2026-11-03 17:31:46.587589043 +0000 UTC m=+849.042278046" Nov 03 17:31:47.016615 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:47.016573 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:48.607782 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:31:48.607745 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:31:55.469093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.468760 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:31:55.479397 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.479368 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.533304 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.533269 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:31:55.579266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579237 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579290 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbpn\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579321 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579490 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579568 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579556 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579586 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579652 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579642 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579669 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.579722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.579705 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.680858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.680820 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbpn\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.680858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.680875 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.680905 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.680944 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.680987 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681033 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681079 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681125 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681213 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681306 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.681570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.681395 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.682153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.682127 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.682300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.682222 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.682649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.682626 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.683669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.683637 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.684447 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.684385 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.743818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.743744 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbpn\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.743949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.743856 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:55.796517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:55.796489 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:56.015922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.015832 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:56.029261 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.029233 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:56.215345 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:31:56.215313 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e36f1e2_8fb5_424d_842f_b3fa89072f0a.slice/crio-7a4d0fb15001cc172ae8debe11d9ef25804e38539c5db4d9a012f2e75a6c23a4 WatchSource:0}: Error finding container 7a4d0fb15001cc172ae8debe11d9ef25804e38539c5db4d9a012f2e75a6c23a4: Status 404 returned error can't find the container with id 7a4d0fb15001cc172ae8debe11d9ef25804e38539c5db4d9a012f2e75a6c23a4 Nov 03 17:31:56.218107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.218082 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:31:56.218265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.218240 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:56.218324 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.218295 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:56.218376 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.218324 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:31:56.582138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.577519 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" event={"ID":"9e36f1e2-8fb5-424d-842f-b3fa89072f0a","Type":"ContainerStarted","Data":"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879"} Nov 03 17:31:56.582138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.577578 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" event={"ID":"9e36f1e2-8fb5-424d-842f-b3fa89072f0a","Type":"ContainerStarted","Data":"7a4d0fb15001cc172ae8debe11d9ef25804e38539c5db4d9a012f2e75a6c23a4"} Nov 03 17:31:56.590379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.590339 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:31:56.701710 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.701655 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podStartSLOduration=1.701637719 podStartE2EDuration="1.701637719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:56.700812419 +0000 UTC m=+859.155501442" watchObservedRunningTime="2026-11-03 17:31:56.701637719 +0000 UTC m=+859.156326747" Nov 03 17:31:56.797518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:56.797490 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:31:57.798758 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:57.798728 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:31:57.799183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:57.798783 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:31:58.797405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:58.797373 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:31:58.797579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:58.797429 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:31:59.797005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:59.796965 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:31:59.797434 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:31:59.797037 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:32:00.796934 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:00.796900 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:32:00.797139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:00.796960 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:32:01.797091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:01.797039 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:32:01.797265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:01.797117 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:32:02.797585 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:02.797552 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:32:02.798145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:02.797604 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 17:32:04.124574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:04.124537 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:32:04.125125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:04.124943 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:32:04.126025 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:04.126000 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:32:08.014111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.014055 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:32:08.014524 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.014351 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" podUID="220de158-6794-4079-ac22-7969ed4a4699" containerName="maas-api" containerID="cri-o://3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984" gracePeriod=30 Nov 03 17:32:08.415549 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.415521 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:32:08.465256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465224 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\") pod \"220de158-6794-4079-ac22-7969ed4a4699\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " Nov 03 17:32:08.465439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465341 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\") pod \"220de158-6794-4079-ac22-7969ed4a4699\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " Nov 03 17:32:08.465439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465376 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\" (UniqueName: \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\") pod \"220de158-6794-4079-ac22-7969ed4a4699\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " Nov 03 17:32:08.465564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465449 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\") pod \"220de158-6794-4079-ac22-7969ed4a4699\" (UID: \"220de158-6794-4079-ac22-7969ed4a4699\") " Nov 03 17:32:08.465729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465642 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca" pod "220de158-6794-4079-ac22-7969ed4a4699" (UID: "220de158-6794-4079-ac22-7969ed4a4699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:08.465873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.465814 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220de158-6794-4079-ac22-7969ed4a4699-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:08.467698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.467675 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls" pod "220de158-6794-4079-ac22-7969ed4a4699" (UID: "220de158-6794-4079-ac22-7969ed4a4699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:08.467797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.467723 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl" pod "220de158-6794-4079-ac22-7969ed4a4699" (UID: "220de158-6794-4079-ac22-7969ed4a4699"). InnerVolumeSpecName "kube-api-access-sf9nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:08.467797 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.467764 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs" pod "220de158-6794-4079-ac22-7969ed4a4699" (UID: "220de158-6794-4079-ac22-7969ed4a4699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:08.574230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.574139 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:08.574230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.574181 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9nl\" (UniqueName: \"kubernetes.io/projected/220de158-6794-4079-ac22-7969ed4a4699-kube-api-access-sf9nl\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:08.574230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.574198 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220de158-6794-4079-ac22-7969ed4a4699-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:08.697579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.697491 4885 generic.go:356] "Generic (PLEG): container finished" podID="220de158-6794-4079-ac22-7969ed4a4699" containerID="3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984" exitCode=0 Nov 03 17:32:08.697852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.697539 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" Nov 03 17:32:08.697938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.697560 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" event={"ID":"220de158-6794-4079-ac22-7969ed4a4699","Type":"ContainerDied","Data":"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984"} Nov 03 17:32:08.697999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.697963 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h" event={"ID":"220de158-6794-4079-ac22-7969ed4a4699","Type":"ContainerDied","Data":"dea123ecaa8ba545b3975f8c10c350249693b4553f774db6247fa65cc0d43cb9"} Nov 03 17:32:08.698053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.698005 4885 scope.go:122] "RemoveContainer" containerID="3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984" Nov 03 17:32:08.711127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.711105 4885 scope.go:122] "RemoveContainer" containerID="3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984" Nov 03 17:32:08.711435 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:32:08.711412 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984\": container with ID starting with 3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984 not found: ID does not exist" containerID="3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984" Nov 03 17:32:08.711526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.711447 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984"} err="failed to get container status \"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984\": rpc error: code = NotFound desc = could not find container \"3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984\": container with ID starting with 3cdc88f73d1cae22e8ea3a888058923699d1bd964bf53c1028cec672cec7f984 not found: ID does not exist" Nov 03 17:32:08.890032 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.889996 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:32:08.893987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:08.893856 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fc5ec-658bb8cb6c-rkx5h"] Nov 03 17:32:09.661085 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:09.661048 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220de158-6794-4079-ac22-7969ed4a4699" path="/var/lib/kubelet/pods/220de158-6794-4079-ac22-7969ed4a4699/volumes" Nov 03 17:32:11.008366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.008327 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:32:11.008930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.008910 4885 cpu_manager.go:414] "containerMap: removing container" podUID="220de158-6794-4079-ac22-7969ed4a4699" containerName="maas-api" Nov 03 17:32:11.008977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.008937 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220de158-6794-4079-ac22-7969ed4a4699" containerName="maas-api" Nov 03 17:32:11.009117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.009101 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="220de158-6794-4079-ac22-7969ed4a4699" containerName="maas-api" Nov 03 17:32:11.013529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.013509 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.071542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.071503 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:32:11.144606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.144568 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:32:11.246990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.246960 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247022 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247179 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247230 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cv6\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247296 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247327 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247530 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247359 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.247530 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.247419 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348309 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348366 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348398 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348422 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cv6\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348461 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348511 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348545 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348582 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348606 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.348976 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348943 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.349145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.348997 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.349145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.349056 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.349145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.349097 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.349452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.349431 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.350983 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.350962 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.351244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.351225 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.413158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.413128 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.413313 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.413195 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cv6\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\") pod \"e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:11.626810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:11.626777 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:12.097945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.097864 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:32:12.139722 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:32:12.139691 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10776004_6223_4825_8ac7_edf61a79a43b.slice/crio-01ed7060a484b6dc96bdf967b4b8538f4c7225f7c2eccf7d8a1c7b2233d9b940 WatchSource:0}: Error finding container 01ed7060a484b6dc96bdf967b4b8538f4c7225f7c2eccf7d8a1c7b2233d9b940: Status 404 returned error can't find the container with id 01ed7060a484b6dc96bdf967b4b8538f4c7225f7c2eccf7d8a1c7b2233d9b940 Nov 03 17:32:12.148818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.148777 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:12.149026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.149003 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:12.149209 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.149186 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:12.753904 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.753859 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" event={"ID":"10776004-6223-4825-8ac7-edf61a79a43b","Type":"ContainerStarted","Data":"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb"} Nov 03 17:32:12.754224 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.754198 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" event={"ID":"10776004-6223-4825-8ac7-edf61a79a43b","Type":"ContainerStarted","Data":"01ed7060a484b6dc96bdf967b4b8538f4c7225f7c2eccf7d8a1c7b2233d9b940"} Nov 03 17:32:12.879641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:12.879576 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podStartSLOduration=2.879556465 podStartE2EDuration="2.879556465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:12.877689282 +0000 UTC m=+875.332378311" watchObservedRunningTime="2026-11-03 17:32:12.879556465 +0000 UTC m=+875.334245491" Nov 03 17:32:13.628894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:13.628862 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:13.629614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:13.629589 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:13.629774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:13.629752 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:14.638498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:14.638459 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:14.639142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:14.639108 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:15.112871 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.112781 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:32:15.139545 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.139512 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.159892 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.159945 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160017 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160136 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160173 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjsh\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160221 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160290 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160323 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.173113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.160360 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.195894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.195715 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-147834-openshift-default-dockercfg-2pfbf\"" Nov 03 17:32:15.252756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.249327 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:32:15.260979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.260944 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.260979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.260978 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261342 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261267 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261342 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261322 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261344 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjsh\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261374 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261415 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261418 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261436 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261456 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.261769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.261695 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.262336 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.262312 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.266399 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.266370 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.266525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.262708 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.266525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.265567 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.266525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.262457 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.330869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.330787 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.336600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.336570 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjsh\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.454772 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.454737 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:15.632534 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.632491 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:15.632738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.632557 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:15.921605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.920857 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:15.921605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.920945 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:15.921605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.920993 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:32:15.931417 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:15.930479 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:32:16.634092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:16.633466 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:16.634092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:16.633532 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:16.799533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:16.799498 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" event={"ID":"49f36c0e-3505-4dfb-8c4f-03e8eae7089e","Type":"ContainerStarted","Data":"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af"} Nov 03 17:32:16.799706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:16.799590 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" event={"ID":"49f36c0e-3505-4dfb-8c4f-03e8eae7089e","Type":"ContainerStarted","Data":"86de936dff38a7b2dec3058a1c50258465708a70753951e8f1e3be293470424f"} Nov 03 17:32:17.047132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.046991 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podStartSLOduration=2.046972124 podStartE2EDuration="2.046972124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:16.979271942 +0000 UTC m=+879.433960966" watchObservedRunningTime="2026-11-03 17:32:17.046972124 +0000 UTC m=+879.501661150" Nov 03 17:32:17.461087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.455400 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:17.470091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.468248 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 17:32:17.470091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.468316 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 17:32:17.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.628461 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:17.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.628538 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:17.798467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:17.798387 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:32:18.456117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:18.456057 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 17:32:18.456601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:18.456163 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 17:32:18.627885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:18.627447 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:32:18.627885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:18.627508 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 03 17:32:18.654755 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:32:18.654712 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:32:19.455553 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:19.455517 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 17:32:19.455717 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:19.455570 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 17:32:19.742638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:19.742564 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:19.743660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:19.743600 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:19.746172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:19.746130 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:32:20.455641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:20.455604 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 17:32:20.455827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:20.455672 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 17:32:21.455252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:21.455217 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Nov 03 17:32:21.455627 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:21.455275 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Nov 03 17:32:22.615709 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:22.615680 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:22.616575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:22.616549 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:22.617758 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:22.617731 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:32:32.897299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:32.897256 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" podUID="82e0178e-339d-4632-a606-431ac81f932c" containerName="main" containerID="cri-o://ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8" gracePeriod=45 Nov 03 17:32:33.058549 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.058509 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" podUID="82e0178e-339d-4632-a606-431ac81f932c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused" Nov 03 17:32:33.166643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.166621 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:32:33.284601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284561 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.284773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284615 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.284773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284712 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.284896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284795 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\" (UniqueName: \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.284896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284843 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:33.284896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284849 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.285054 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.284965 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\" (UniqueName: \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\") pod \"82e0178e-339d-4632-a606-431ac81f932c\" (UID: \"82e0178e-339d-4632-a606-431ac81f932c\") " Nov 03 17:32:33.285134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.285090 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:33.285450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.285422 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:33.285563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.285508 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.285563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.285527 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.287844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.287820 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:33.288109 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.288052 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "kube-api-access-mdrhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:33.288289 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.288272 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm" pod "82e0178e-339d-4632-a606-431ac81f932c" (UID: "82e0178e-339d-4632-a606-431ac81f932c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:33.386538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.386507 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.386538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.386539 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82e0178e-339d-4632-a606-431ac81f932c-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.386725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.386552 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82e0178e-339d-4632-a606-431ac81f932c-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.386725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.386565 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdrhk\" (UniqueName: \"kubernetes.io/projected/82e0178e-339d-4632-a606-431ac81f932c-kube-api-access-mdrhk\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:32:33.948723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.948629 4885 generic.go:356] "Generic (PLEG): container finished" podID="82e0178e-339d-4632-a606-431ac81f932c" containerID="ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8" exitCode=0 Nov 03 17:32:33.948723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.948680 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" Nov 03 17:32:33.949254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.948724 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" event={"ID":"82e0178e-339d-4632-a606-431ac81f932c","Type":"ContainerDied","Data":"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8"} Nov 03 17:32:33.949254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.948778 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2" event={"ID":"82e0178e-339d-4632-a606-431ac81f932c","Type":"ContainerDied","Data":"e929e07a9218848a5a7d7cb2f21a62a624fee2695ec1ecf54321d678d283a808"} Nov 03 17:32:33.949254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.948800 4885 scope.go:122] "RemoveContainer" containerID="ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8" Nov 03 17:32:33.957275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.957257 4885 scope.go:122] "RemoveContainer" containerID="ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8" Nov 03 17:32:33.957539 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:32:33.957519 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8\": container with ID starting with ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8 not found: ID does not exist" containerID="ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8" Nov 03 17:32:33.957601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:33.957551 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8"} err="failed to get container status \"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8\": rpc error: code = NotFound desc = could not find container \"ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8\": container with ID starting with ad1db87afe0a739562c5e562a7b5886dd03ef822d4c5115537f407489905c2b8 not found: ID does not exist" Nov 03 17:32:34.119321 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:34.119288 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:32:34.121985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:34.121964 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/e2e-conflict-a-0ff8cfd1-kserve-556b94d657-mchs2"] Nov 03 17:32:35.661543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:35.661512 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e0178e-339d-4632-a606-431ac81f932c" path="/var/lib/kubelet/pods/82e0178e-339d-4632-a606-431ac81f932c/volumes" Nov 03 17:32:35.856471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:35.856433 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:32:48.605976 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:32:48.605940 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:32:54.666325 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:32:54.666292 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:33:11.096609 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.090693 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:33:11.109815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.109778 4885 cpu_manager.go:414] "containerMap: removing container" podUID="82e0178e-339d-4632-a606-431ac81f932c" containerName="main" Nov 03 17:33:11.111018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.110996 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e0178e-339d-4632-a606-431ac81f932c" containerName="main" Nov 03 17:33:11.111358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.111342 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e0178e-339d-4632-a606-431ac81f932c" containerName="main" Nov 03 17:33:11.115596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.115574 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.143112 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.143059 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.143390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.143372 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.143557 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.143542 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.145491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.145465 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.145673 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.145657 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.145823 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.145797 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fsp\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.145942 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.145926 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.146056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.146039 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.146165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.146109 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.188385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.188354 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:33:11.247636 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247600 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247679 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247707 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fsp\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247751 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247814 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247847 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247904 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247926 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.247960 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248771 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.248460 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.248960 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.248937 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.249295 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.249187 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.249295 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.249245 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.249482 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.249350 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.250697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.250673 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.253300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.253274 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.311496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.311467 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.311651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.311619 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fsp\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\") pod \"e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.443422 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.443390 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:11.868909 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:33:11.868875 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f474b0f_5891_42e2_8bff_d7b3217aa547.slice/crio-48e3f14e4471f26093da43db4bd701cc0e7e8f1c732cd8d1b7a8c28048d48b71 WatchSource:0}: Error finding container 48e3f14e4471f26093da43db4bd701cc0e7e8f1c732cd8d1b7a8c28048d48b71: Status 404 returned error can't find the container with id 48e3f14e4471f26093da43db4bd701cc0e7e8f1c732cd8d1b7a8c28048d48b71 Nov 03 17:33:11.871156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.871131 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:33:11.871702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.871672 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:11.871751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.871739 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:11.871791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:11.871767 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:12.230037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:12.230004 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" event={"ID":"9f474b0f-5891-42e2-8bff-d7b3217aa547","Type":"ContainerStarted","Data":"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f"} Nov 03 17:33:12.230560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:12.230080 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" event={"ID":"9f474b0f-5891-42e2-8bff-d7b3217aa547","Type":"ContainerStarted","Data":"48e3f14e4471f26093da43db4bd701cc0e7e8f1c732cd8d1b7a8c28048d48b71"} Nov 03 17:33:12.356343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:12.356285 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podStartSLOduration=1.356268051 podStartE2EDuration="1.356268051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:12.353447974 +0000 UTC m=+934.808137001" watchObservedRunningTime="2026-11-03 17:33:12.356268051 +0000 UTC m=+934.810957078" Nov 03 17:33:12.448251 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:12.448166 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:13.446144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:13.446112 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:13.446650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:13.446171 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:14.443783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:14.443747 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:14.443967 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:14.443819 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:14.974050 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:14.974009 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs"] Nov 03 17:33:15.071662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.071628 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs"] Nov 03 17:33:15.071987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.071967 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.130093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.128001 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:33:15.147348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.147306 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.147637 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.147617 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlrs\" (UniqueName: \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.147902 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.147887 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.178759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.178701 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:33:15.178919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.178840 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:33:15.178987 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.178967 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:33:15.249494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.249400 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.249494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.249451 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlrs\" (UniqueName: \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.249678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.249569 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.250337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.250313 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.252266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.252239 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.332749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.332712 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlrs\" (UniqueName: \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\") pod \"maas-api-revoke-sub-3ef76450-bhvgs\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.395284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.395245 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:15.443868 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.443835 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:15.444130 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.444101 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:15.818748 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:15.818710 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs"] Nov 03 17:33:16.255297 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:16.255242 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" event={"ID":"efbfcfa1-8abb-40e5-912f-dd66fe607d91","Type":"ContainerStarted","Data":"dd1dd7fe780776555a6f4d79821528de5fa83304c4f2a6c7a97a294998f37e79"} Nov 03 17:33:16.255736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:16.255717 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" event={"ID":"efbfcfa1-8abb-40e5-912f-dd66fe607d91","Type":"ContainerStarted","Data":"bb6ac21adcf51c6d5d3ae3d1526d7a49ce1f64f209f0361cb09707aa3a3dd527"} Nov 03 17:33:16.380923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:16.380875 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" podStartSLOduration=2.380862651 podStartE2EDuration="2.380862651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:16.37961651 +0000 UTC m=+938.834305536" watchObservedRunningTime="2026-11-03 17:33:16.380862651 +0000 UTC m=+938.835551676" Nov 03 17:33:16.445978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:16.445583 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:16.445978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:16.445631 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:17.262824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:17.262728 4885 generic.go:356] "Generic (PLEG): container finished" podID="efbfcfa1-8abb-40e5-912f-dd66fe607d91" containerID="dd1dd7fe780776555a6f4d79821528de5fa83304c4f2a6c7a97a294998f37e79" exitCode=0 Nov 03 17:33:17.263388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:17.262988 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" event={"ID":"efbfcfa1-8abb-40e5-912f-dd66fe607d91","Type":"ContainerDied","Data":"dd1dd7fe780776555a6f4d79821528de5fa83304c4f2a6c7a97a294998f37e79"} Nov 03 17:33:17.444701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:17.444666 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:17.444889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:17.444722 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:18.414810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.414778 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:18.443835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.443803 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 03 17:33:18.443992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.443859 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 03 17:33:18.494213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.494178 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\") pod \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " Nov 03 17:33:18.494385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.494230 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\" (UniqueName: \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\") pod \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " Nov 03 17:33:18.494385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.494316 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\") pod \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\" (UID: \"efbfcfa1-8abb-40e5-912f-dd66fe607d91\") " Nov 03 17:33:18.494530 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.494505 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca" pod "efbfcfa1-8abb-40e5-912f-dd66fe607d91" (UID: "efbfcfa1-8abb-40e5-912f-dd66fe607d91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:33:18.494884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.494863 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:18.496560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.496536 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs" pod "efbfcfa1-8abb-40e5-912f-dd66fe607d91" (UID: "efbfcfa1-8abb-40e5-912f-dd66fe607d91"). InnerVolumeSpecName "kube-api-access-cvlrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:18.496632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.496574 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca" pod "efbfcfa1-8abb-40e5-912f-dd66fe607d91" (UID: "efbfcfa1-8abb-40e5-912f-dd66fe607d91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:33:18.595918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.595831 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvlrs\" (UniqueName: \"kubernetes.io/projected/efbfcfa1-8abb-40e5-912f-dd66fe607d91-kube-api-access-cvlrs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:18.595918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:18.595869 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efbfcfa1-8abb-40e5-912f-dd66fe607d91-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:18.606313 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:33:18.606277 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:33:19.274658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.274627 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" event={"ID":"efbfcfa1-8abb-40e5-912f-dd66fe607d91","Type":"ContainerDied","Data":"bb6ac21adcf51c6d5d3ae3d1526d7a49ce1f64f209f0361cb09707aa3a3dd527"} Nov 03 17:33:19.274658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.274646 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs" Nov 03 17:33:19.274923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.274672 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb6ac21adcf51c6d5d3ae3d1526d7a49ce1f64f209f0361cb09707aa3a3dd527" Nov 03 17:33:19.449277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.449249 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:19.449747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.449728 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:19.450502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:19.450483 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:33:20.588401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.588363 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:33:20.588831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.588816 4885 cpu_manager.go:414] "containerMap: removing container" podUID="efbfcfa1-8abb-40e5-912f-dd66fe607d91" containerName="revoke-keys" Nov 03 17:33:20.588873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.588833 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbfcfa1-8abb-40e5-912f-dd66fe607d91" containerName="revoke-keys" Nov 03 17:33:20.588909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.588899 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbfcfa1-8abb-40e5-912f-dd66fe607d91" containerName="revoke-keys" Nov 03 17:33:20.592000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.591981 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.647739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.647705 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:33:20.693875 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.693850 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e34dd2\"/\"kube-root-ca.crt\"" Nov 03 17:33:20.695399 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.695382 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e34dd2\"/\"openshift-service-ca.crt\"" Nov 03 17:33:20.695619 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.695608 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e34dd2\"/\"test-model-e34dd2-kserve-self-signed-certs\"" Nov 03 17:33:20.699504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.699490 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e34dd2\"/\"default-dockercfg-4qtvk\"" Nov 03 17:33:20.722007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.721979 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.722178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.722013 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.722178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.722111 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.722287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.722242 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.722287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.722277 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.722389 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.722332 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mzx\" (UniqueName: \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823040 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823119 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823150 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823179 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mzx\" (UniqueName: \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823235 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823256 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823565 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823529 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823569 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.823722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.823696 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.825659 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.825636 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.825924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.825908 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.894114 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.894081 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mzx\" (UniqueName: \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\") pod \"test-model-e34dd2-kserve-76889756b7-v8l4s\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:20.903485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:20.903452 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:21.285716 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:33:21.285687 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12db0d7c_ef44_4fe5_a887_79740a5f3261.slice/crio-7aa33591bd4c6e798632132e2569f82becda002b97e72aa563d7aed4c0a8cae1 WatchSource:0}: Error finding container 7aa33591bd4c6e798632132e2569f82becda002b97e72aa563d7aed4c0a8cae1: Status 404 returned error can't find the container with id 7aa33591bd4c6e798632132e2569f82becda002b97e72aa563d7aed4c0a8cae1 Nov 03 17:33:21.287722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:21.287703 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:33:22.293308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:22.293263 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" event={"ID":"12db0d7c-ef44-4fe5-a887-79740a5f3261","Type":"ContainerStarted","Data":"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20"} Nov 03 17:33:22.293308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:22.293308 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" event={"ID":"12db0d7c-ef44-4fe5-a887-79740a5f3261","Type":"ContainerStarted","Data":"7aa33591bd4c6e798632132e2569f82becda002b97e72aa563d7aed4c0a8cae1"} Nov 03 17:33:22.415375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:22.415323 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" podStartSLOduration=2.415311714 podStartE2EDuration="2.415311714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:22.414583214 +0000 UTC m=+944.869272237" watchObservedRunningTime="2026-11-03 17:33:22.415311714 +0000 UTC m=+944.870000737" Nov 03 17:33:22.904578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:22.904544 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:23.459956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.459912 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:33:23.469608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.469561 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.527138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.526444 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3f1cf1-openshift-default-dockercfg-zr9s2\"" Nov 03 17:33:23.564378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564347 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564503 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564546 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564579 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xscjh\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564676 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564701 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564780 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564809 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.564911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.564845 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.581985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.581955 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:33:23.665529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665503 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665535 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665570 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665600 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665657 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665742 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665827 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665853 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.665926 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665884 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xscjh\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.666143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.665924 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.666287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.666264 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.666388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.666298 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.666388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.666372 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.666479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.666395 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.668149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.668132 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.668242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.668226 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.729302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.729240 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.729405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.729317 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xscjh\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\") pod \"e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:23.783314 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:23.783290 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:24.215819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:24.215783 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:33:24.220118 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:33:24.220043 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c55830_f20b_4846_a83f_1ad9d5249322.slice/crio-ee67b5e42d1fbb9d5478e47bfb7ca0528a265334ba0300711fae9e3c12e0efa6 WatchSource:0}: Error finding container ee67b5e42d1fbb9d5478e47bfb7ca0528a265334ba0300711fae9e3c12e0efa6: Status 404 returned error can't find the container with id ee67b5e42d1fbb9d5478e47bfb7ca0528a265334ba0300711fae9e3c12e0efa6 Nov 03 17:33:24.227563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:24.227525 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:24.227676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:24.227615 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:24.227676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:24.227662 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:33:24.365110 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:24.364310 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" event={"ID":"63c55830-f20b-4846-a83f-1ad9d5249322","Type":"ContainerStarted","Data":"ee67b5e42d1fbb9d5478e47bfb7ca0528a265334ba0300711fae9e3c12e0efa6"} Nov 03 17:33:25.370741 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:25.370701 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" event={"ID":"63c55830-f20b-4846-a83f-1ad9d5249322","Type":"ContainerStarted","Data":"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795"} Nov 03 17:33:25.546049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:25.546000 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podStartSLOduration=2.545984673 podStartE2EDuration="2.545984673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:25.544999316 +0000 UTC m=+947.999688341" watchObservedRunningTime="2026-11-03 17:33:25.545984673 +0000 UTC m=+948.000673697" Nov 03 17:33:25.783614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:25.783523 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:25.785181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:25.785151 4885 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 17:33:25.785305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:25.785200 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 17:33:26.784345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:26.784296 4885 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 17:33:26.784831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:26.784358 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 17:33:27.784746 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:27.784666 4885 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 17:33:27.784746 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:27.784717 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 17:33:28.783827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:28.783788 4885 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 17:33:28.783995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:28.783853 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 17:33:29.784715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:29.784681 4885 patch_prober.go:29] interesting pod/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Nov 03 17:33:29.785122 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:29.784739 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Nov 03 17:33:30.885108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.885047 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:30.886745 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.886005 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:30.887015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.886994 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:33:30.904379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.904353 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:30.917492 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.917470 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:30.997881 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:30.997848 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:33:32.132600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:32.132573 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:33:45.103168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:45.103134 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:33:45.103738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:45.103361 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" containerID="cri-o://9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e" gracePeriod=30 Nov 03 17:33:48.692370 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:33:48.692334 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:33:50.378327 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.378305 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:33:50.408238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408209 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408305 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408340 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408371 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408501 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408411 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408499 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408572 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.408949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.408900 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:33:50.409113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409033 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:33:50.409113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409083 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.409263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409144 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\") pod \"17616578-c382-488d-8b91-adfe63a684b5\" (UID: \"17616578-c382-488d-8b91-adfe63a684b5\") " Nov 03 17:33:50.409662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409636 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17616578-c382-488d-8b91-adfe63a684b5-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.409755 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409668 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.410322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.409961 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:33:50.410322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.410135 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:33:50.410322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.410296 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:33:50.411148 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.411123 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:33:50.411326 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.411303 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:50.411511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.411493 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "kube-api-access-x5s9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:50.412519 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.412484 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy" pod "17616578-c382-488d-8b91-adfe63a684b5" (UID: "17616578-c382-488d-8b91-adfe63a684b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:33:50.510863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510824 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17616578-c382-488d-8b91-adfe63a684b5-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.510863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510860 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.511152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510875 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.511152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510888 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5s9s\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-kube-api-access-x5s9s\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.511152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510901 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17616578-c382-488d-8b91-adfe63a684b5-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.511152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510914 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.511152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.510926 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17616578-c382-488d-8b91-adfe63a684b5-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:33:50.678888 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.678804 4885 generic.go:356] "Generic (PLEG): container finished" podID="17616578-c382-488d-8b91-adfe63a684b5" containerID="9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e" exitCode=0 Nov 03 17:33:50.678888 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.678850 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" Nov 03 17:33:50.679088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.678853 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" event={"ID":"17616578-c382-488d-8b91-adfe63a684b5","Type":"ContainerDied","Data":"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e"} Nov 03 17:33:50.679088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.678958 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt" event={"ID":"17616578-c382-488d-8b91-adfe63a684b5","Type":"ContainerDied","Data":"1b080a2b69a975292d5ba06db71fbf8b57ce2e2bd8a267e755b4c1d95e4ea5bd"} Nov 03 17:33:50.679088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.678980 4885 scope.go:122] "RemoveContainer" containerID="9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e" Nov 03 17:33:50.688371 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.688350 4885 scope.go:122] "RemoveContainer" containerID="9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e" Nov 03 17:33:50.688652 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:33:50.688633 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e\": container with ID starting with 9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e not found: ID does not exist" containerID="9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e" Nov 03 17:33:50.688697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.688661 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e"} err="failed to get container status \"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e\": rpc error: code = NotFound desc = could not find container \"9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e\": container with ID starting with 9b3687d3f0338c3137d418c686b3f7166029972581d7d1b6ead5b5dfb67be23e not found: ID does not exist" Nov 03 17:33:50.858058 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.857542 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:33:50.866512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:50.866480 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-swxpt"] Nov 03 17:33:51.661883 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:33:51.661841 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17616578-c382-488d-8b91-adfe63a684b5" path="/var/lib/kubelet/pods/17616578-c382-488d-8b91-adfe63a684b5/volumes" Nov 03 17:34:05.437712 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:05.437669 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:34:18.607153 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:34:18.607114 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:34:25.574419 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.574368 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx"] Nov 03 17:34:25.575116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.575052 4885 cpu_manager.go:414] "containerMap: removing container" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" Nov 03 17:34:25.575116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.575097 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" Nov 03 17:34:25.575350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.575237 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="17616578-c382-488d-8b91-adfe63a684b5" containerName="istio-proxy" Nov 03 17:34:25.579350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.579328 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.633118 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.633051 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:34:25.684502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.684465 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:34:25.684709 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.684524 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:34:25.684709 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.684552 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:34:25.686723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.686693 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx"] Nov 03 17:34:25.687175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.687151 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mks\" (UniqueName: \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.687285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.687275 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.687380 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.687366 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.787940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.787909 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.787940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.787944 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mks\" (UniqueName: \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.788266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.788101 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.788814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.788790 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.790507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.790478 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.850904 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.850831 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mks\" (UniqueName: \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\") pod \"maas-api-revoke-sub-4974004a-r89sx\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:25.896226 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:25.896199 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:26.267624 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:34:26.267587 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3002c784_a018_4389_bba1_e76fe9f21275.slice/crio-ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a WatchSource:0}: Error finding container ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a: Status 404 returned error can't find the container with id ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a Nov 03 17:34:26.269743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:26.269718 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx"] Nov 03 17:34:26.901998 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:26.901955 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" event={"ID":"3002c784-a018-4389-bba1-e76fe9f21275","Type":"ContainerStarted","Data":"5cef0f7cead88bf431bc022345c9b7d2e721c4892922b8f57a944925c84a4877"} Nov 03 17:34:26.902517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:26.902036 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" event={"ID":"3002c784-a018-4389-bba1-e76fe9f21275","Type":"ContainerStarted","Data":"ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a"} Nov 03 17:34:27.072291 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:27.072240 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" podStartSLOduration=2.072225987 podStartE2EDuration="2.072225987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:27.020362646 +0000 UTC m=+1009.475051669" watchObservedRunningTime="2026-11-03 17:34:27.072225987 +0000 UTC m=+1009.526915010" Nov 03 17:34:27.908800 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:27.908770 4885 generic.go:356] "Generic (PLEG): container finished" podID="3002c784-a018-4389-bba1-e76fe9f21275" containerID="5cef0f7cead88bf431bc022345c9b7d2e721c4892922b8f57a944925c84a4877" exitCode=0 Nov 03 17:34:27.909212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:27.908832 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" event={"ID":"3002c784-a018-4389-bba1-e76fe9f21275","Type":"ContainerDied","Data":"5cef0f7cead88bf431bc022345c9b7d2e721c4892922b8f57a944925c84a4877"} Nov 03 17:34:29.065281 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.065253 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:29.123598 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.123576 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\") pod \"3002c784-a018-4389-bba1-e76fe9f21275\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " Nov 03 17:34:29.123756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.123638 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\" (UniqueName: \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\") pod \"3002c784-a018-4389-bba1-e76fe9f21275\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " Nov 03 17:34:29.123756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.123691 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\") pod \"3002c784-a018-4389-bba1-e76fe9f21275\" (UID: \"3002c784-a018-4389-bba1-e76fe9f21275\") " Nov 03 17:34:29.124000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.123974 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca" pod "3002c784-a018-4389-bba1-e76fe9f21275" (UID: "3002c784-a018-4389-bba1-e76fe9f21275"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:29.125892 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.125860 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks" pod "3002c784-a018-4389-bba1-e76fe9f21275" (UID: "3002c784-a018-4389-bba1-e76fe9f21275"). InnerVolumeSpecName "kube-api-access-r8mks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:29.126011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.125995 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca" pod "3002c784-a018-4389-bba1-e76fe9f21275" (UID: "3002c784-a018-4389-bba1-e76fe9f21275"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:29.225246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.225166 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:29.225246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.225200 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3002c784-a018-4389-bba1-e76fe9f21275-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:29.225246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.225217 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8mks\" (UniqueName: \"kubernetes.io/projected/3002c784-a018-4389-bba1-e76fe9f21275-kube-api-access-r8mks\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:29.920452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.920361 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" event={"ID":"3002c784-a018-4389-bba1-e76fe9f21275","Type":"ContainerDied","Data":"ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a"} Nov 03 17:34:29.920452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.920418 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec0fd2f66db801c5127dc3023be4d6c74eb7520eacf6c261f7c3026bbf22650a" Nov 03 17:34:29.920452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:29.920417 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx" Nov 03 17:34:31.707319 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:31.707286 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:34:34.380089 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:34.379166 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:35.070915 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:35.070874 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:35.071254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:35.071208 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:35.101899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:35.101783 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": context deadline exceeded" Nov 03 17:34:35.183090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:35.182219 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:35.183090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:35.182301 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:44.432736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.432695 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv"] Nov 03 17:34:44.433348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.433329 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3002c784-a018-4389-bba1-e76fe9f21275" containerName="revoke-keys" Nov 03 17:34:44.433431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.433351 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3002c784-a018-4389-bba1-e76fe9f21275" containerName="revoke-keys" Nov 03 17:34:44.433502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.433490 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3002c784-a018-4389-bba1-e76fe9f21275" containerName="revoke-keys" Nov 03 17:34:44.436548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.436526 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.491250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.491219 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv"] Nov 03 17:34:44.539363 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.539337 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:34:44.539363 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.539359 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:34:44.539363 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.539369 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:34:44.541930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.541907 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:34:44.582913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.582865 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9sh\" (UniqueName: \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.583078 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.583038 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.583144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.583100 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.684574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.684480 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.684574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.684554 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9sh\" (UniqueName: \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.684769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.684630 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.685161 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.685133 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.687253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.687233 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.747693 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.747659 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9sh\" (UniqueName: \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\") pod \"maas-api-revoke-sub-c73f7358-8ncjv\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:44.749382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:44.749365 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:45.164796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:45.164757 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv"] Nov 03 17:34:46.073108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:46.073060 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" event={"ID":"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be","Type":"ContainerStarted","Data":"371df12697f3b548926b977acf29c6919068a8f7d25e2aa56445cd49b411230e"} Nov 03 17:34:46.073523 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:46.073128 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" event={"ID":"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be","Type":"ContainerStarted","Data":"cc376d4f272b8a14e70c3a0cc9c5b4b3a8cd1548184e1d40ccf8e50af5313081"} Nov 03 17:34:46.196023 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:46.195978 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" podStartSLOduration=2.195961721 podStartE2EDuration="2.195961721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:46.194417045 +0000 UTC m=+1028.649106066" watchObservedRunningTime="2026-11-03 17:34:46.195961721 +0000 UTC m=+1028.650650745" Nov 03 17:34:47.079689 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:47.079651 4885 generic.go:356] "Generic (PLEG): container finished" podID="a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" containerID="371df12697f3b548926b977acf29c6919068a8f7d25e2aa56445cd49b411230e" exitCode=0 Nov 03 17:34:47.080097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:47.079716 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" event={"ID":"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be","Type":"ContainerDied","Data":"371df12697f3b548926b977acf29c6919068a8f7d25e2aa56445cd49b411230e"} Nov 03 17:34:48.552763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.552468 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:48.670186 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:34:48.669994 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:34:48.680154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.678179 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\") pod \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " Nov 03 17:34:48.680154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.678243 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\") pod \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " Nov 03 17:34:48.680154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.678391 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\" (UniqueName: \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\") pod \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\" (UID: \"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be\") " Nov 03 17:34:48.680154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.679650 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca" pod "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" (UID: "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:48.692088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.690491 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca" pod "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" (UID: "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:48.699301 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.694585 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh" pod "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" (UID: "a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be"). InnerVolumeSpecName "kube-api-access-ms9sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:48.788199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.783006 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:48.788199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.783041 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:48.788199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:48.783057 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9sh\" (UniqueName: \"kubernetes.io/projected/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be-kube-api-access-ms9sh\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.153126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:49.151192 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" event={"ID":"a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be","Type":"ContainerDied","Data":"cc376d4f272b8a14e70c3a0cc9c5b4b3a8cd1548184e1d40ccf8e50af5313081"} Nov 03 17:34:49.153126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:49.151256 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc376d4f272b8a14e70c3a0cc9c5b4b3a8cd1548184e1d40ccf8e50af5313081" Nov 03 17:34:49.153126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:49.151325 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv" Nov 03 17:34:50.069283 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.069239 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:50.069906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.069871 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.070453 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.070430 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:50.070604 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.070584 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.228938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.226492 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:50.228938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.226551 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.369146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.344189 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:34:50.383146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.375740 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.570191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.570148 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 17:34:50.582373 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.582339 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:34:50.582662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.582631 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.599230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.599191 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:34:50.616018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:34:50.615982 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:00.653854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.653818 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq"] Nov 03 17:35:00.654331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.654253 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" containerName="revoke-keys" Nov 03 17:35:00.654331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.654264 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" containerName="revoke-keys" Nov 03 17:35:00.654406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.654353 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" containerName="revoke-keys" Nov 03 17:35:00.657781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.657757 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.711839 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.711811 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq"] Nov 03 17:35:00.712021 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.711905 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:35:00.720802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.720767 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.720949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.720824 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.720949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.720879 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gtt6\" (UniqueName: \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.760396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.760362 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:35:00.760543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.760412 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:35:00.763216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.763200 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:35:00.821647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.821621 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.821804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.821678 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.821804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.821729 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gtt6\" (UniqueName: \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.822507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.822481 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.824168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.824148 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.885123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.885101 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gtt6\" (UniqueName: \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\") pod \"maas-api-revoke-sub-93aa167c-x5wdq\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:00.969897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:00.969811 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:01.351667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:01.351644 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq"] Nov 03 17:35:02.332705 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:02.332669 4885 generic.go:356] "Generic (PLEG): container finished" podID="4ee6a1c8-420f-4466-9c98-a3b151317b40" containerID="44b31a549908c17e4f038185955f4be0e91a03038c67bda24845921640739f13" exitCode=0 Nov 03 17:35:02.333199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:02.332750 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" event={"ID":"4ee6a1c8-420f-4466-9c98-a3b151317b40","Type":"ContainerDied","Data":"44b31a549908c17e4f038185955f4be0e91a03038c67bda24845921640739f13"} Nov 03 17:35:02.333199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:02.332797 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" event={"ID":"4ee6a1c8-420f-4466-9c98-a3b151317b40","Type":"ContainerStarted","Data":"213e5e5af95eccfe5da8244f6caff29e52c659b0e04beb3a80b4e006ac3465e9"} Nov 03 17:35:03.064163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.064127 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:35:03.726116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.707553 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:03.811802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.791544 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\" (UniqueName: \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\") pod \"4ee6a1c8-420f-4466-9c98-a3b151317b40\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " Nov 03 17:35:03.811802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.791725 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\") pod \"4ee6a1c8-420f-4466-9c98-a3b151317b40\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " Nov 03 17:35:03.811802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.791766 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\") pod \"4ee6a1c8-420f-4466-9c98-a3b151317b40\" (UID: \"4ee6a1c8-420f-4466-9c98-a3b151317b40\") " Nov 03 17:35:03.841177 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.812390 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca" pod "4ee6a1c8-420f-4466-9c98-a3b151317b40" (UID: "4ee6a1c8-420f-4466-9c98-a3b151317b40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:03.849311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.849255 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6" pod "4ee6a1c8-420f-4466-9c98-a3b151317b40" (UID: "4ee6a1c8-420f-4466-9c98-a3b151317b40"). InnerVolumeSpecName "kube-api-access-6gtt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:03.918341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.893897 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:03.918341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.893932 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gtt6\" (UniqueName: \"kubernetes.io/projected/4ee6a1c8-420f-4466-9c98-a3b151317b40-kube-api-access-6gtt6\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:03.942266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.942203 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca" pod "4ee6a1c8-420f-4466-9c98-a3b151317b40" (UID: "4ee6a1c8-420f-4466-9c98-a3b151317b40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:04.022131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:03.995059 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee6a1c8-420f-4466-9c98-a3b151317b40-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:04.375095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:04.372513 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" event={"ID":"4ee6a1c8-420f-4466-9c98-a3b151317b40","Type":"ContainerDied","Data":"213e5e5af95eccfe5da8244f6caff29e52c659b0e04beb3a80b4e006ac3465e9"} Nov 03 17:35:04.375095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:04.372603 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="213e5e5af95eccfe5da8244f6caff29e52c659b0e04beb3a80b4e006ac3465e9" Nov 03 17:35:04.375095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:04.372672 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq" Nov 03 17:35:05.387091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.386150 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:05.464985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.464946 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 03 17:35:05.569207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.569164 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:05.587212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.587167 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:05.608204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.608162 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:05.714841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.714723 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" containerID="cri-o://5d4e51424f0cccfb1c763413502dc1962639151517700e01971b35883ad424e2" gracePeriod=45 Nov 03 17:35:05.792212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.792171 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:35:05.792507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:05.792466 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:06.229093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.228157 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:06.389096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.388155 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:06.389096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.388884 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:35:06.470141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.468172 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:06.470141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.468632 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:35:06.616277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.616180 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:06.625091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.616777 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:35:06.625091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.616324 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 17:35:06.625091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.618408 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:35:06.645035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.616275 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 03 17:35:06.645826 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.645800 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:35:06.998101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:06.993752 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:35:07.046099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.028215 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:07.077842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.072592 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:35:07.091581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.078688 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:35:07.100254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.100230 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:35:07.100596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.100569 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": EOF" Nov 03 17:35:07.100793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.100778 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:35:07.186102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.179029 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:35:07.222498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.222454 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused" Nov 03 17:35:07.463030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.462991 4885 generic.go:356] "Generic (PLEG): container finished" podID="1025f336-45aa-4d1c-98c9-33127113b59e" containerID="5d4e51424f0cccfb1c763413502dc1962639151517700e01971b35883ad424e2" exitCode=0 Nov 03 17:35:07.463646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:07.463624 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" event={"ID":"1025f336-45aa-4d1c-98c9-33127113b59e","Type":"ContainerDied","Data":"5d4e51424f0cccfb1c763413502dc1962639151517700e01971b35883ad424e2"} Nov 03 17:35:08.054412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.054383 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:35:08.201818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.201793 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.202174 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202155 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.202392 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202374 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.202509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202253 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:08.202567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202481 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:08.202710 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202694 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.202825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202807 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:08.203571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202918 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.203571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.202970 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\" (UniqueName: \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\") pod \"1025f336-45aa-4d1c-98c9-33127113b59e\" (UID: \"1025f336-45aa-4d1c-98c9-33127113b59e\") " Nov 03 17:35:08.203571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.203331 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.203571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.203350 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.203571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.203362 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.212259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.212221 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "kube-api-access-87t5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:08.212833 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.212749 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:08.215529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.215497 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs" pod "1025f336-45aa-4d1c-98c9-33127113b59e" (UID: "1025f336-45aa-4d1c-98c9-33127113b59e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:08.313270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.306879 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1025f336-45aa-4d1c-98c9-33127113b59e-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.313270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.306915 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1025f336-45aa-4d1c-98c9-33127113b59e-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.313270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.306932 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87t5p\" (UniqueName: \"kubernetes.io/projected/1025f336-45aa-4d1c-98c9-33127113b59e-kube-api-access-87t5p\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:08.483561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.483521 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" event={"ID":"1025f336-45aa-4d1c-98c9-33127113b59e","Type":"ContainerDied","Data":"9dffc10e404a349044e0bccce1b5d7ab3f45bff4a67436767d88ac341cd76f84"} Nov 03 17:35:08.483996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.483585 4885 scope.go:122] "RemoveContainer" containerID="5d4e51424f0cccfb1c763413502dc1962639151517700e01971b35883ad424e2" Nov 03 17:35:08.483996 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.483589 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7" Nov 03 17:35:08.673050 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.673021 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:35:08.732537 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:08.732497 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-af9e3c/auth-test-model-af9e3c-kserve-84cd76dcd5-szkd7"] Nov 03 17:35:09.661178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:09.661148 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" path="/var/lib/kubelet/pods/1025f336-45aa-4d1c-98c9-33127113b59e/volumes" Nov 03 17:35:15.050637 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:15.050603 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr"] Nov 03 17:35:15.102190 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:15.102154 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e0c045-mvzqr"] Nov 03 17:35:15.661199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:15.661166 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00" path="/var/lib/kubelet/pods/34c61a0c-14bb-4e5c-b0e8-d5a58ace2d00/volumes" Nov 03 17:35:18.605451 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:35:18.605415 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:35:21.998243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:21.998205 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 17:35:24.082437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:24.082400 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:35:27.487699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.487664 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:35:27.488277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488159 4885 cpu_manager.go:414] "containerMap: removing container" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" Nov 03 17:35:27.488277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488173 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" Nov 03 17:35:27.488277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488202 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4ee6a1c8-420f-4466-9c98-a3b151317b40" containerName="revoke-keys" Nov 03 17:35:27.488277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488210 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee6a1c8-420f-4466-9c98-a3b151317b40" containerName="revoke-keys" Nov 03 17:35:27.488438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488278 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1025f336-45aa-4d1c-98c9-33127113b59e" containerName="main" Nov 03 17:35:27.488438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.488294 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee6a1c8-420f-4466-9c98-a3b151317b40" containerName="revoke-keys" Nov 03 17:35:27.491630 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.491611 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.551277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.551249 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"distinct-w0-687fb3-kserve-self-signed-certs\"" Nov 03 17:35:27.555010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.554982 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.555158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.555043 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.555222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.555204 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfgp\" (UniqueName: \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.555261 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.555243 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.555301 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.555269 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.555389 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.555357 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.605224 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.605189 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:35:27.658429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.658168 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfgp\" (UniqueName: \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.658708 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.658687 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.658835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.658822 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.658941 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.658930 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.659133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.659119 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.659285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.659273 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.659789 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.659764 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.663147 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.661184 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.663287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.661387 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.665277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.665099 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.666751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.666730 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.726116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.726056 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfgp\" (UniqueName: \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\") pod \"distinct-w0-687fb3-kserve-858b8df8f-k8h99\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:27.804862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:27.804767 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:28.255580 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:35:28.255548 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b796743_59cf_464d_90a7_b1b0226f9748.slice/crio-2d8aba4dada0f6d0e105923119ee96f364f09853ac85f72acbe47866c6715c85 WatchSource:0}: Error finding container 2d8aba4dada0f6d0e105923119ee96f364f09853ac85f72acbe47866c6715c85: Status 404 returned error can't find the container with id 2d8aba4dada0f6d0e105923119ee96f364f09853ac85f72acbe47866c6715c85 Nov 03 17:35:28.258673 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:28.258647 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:35:28.658461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:28.657704 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" event={"ID":"6b796743-59cf-464d-90a7-b1b0226f9748","Type":"ContainerStarted","Data":"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4"} Nov 03 17:35:28.658461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:28.657758 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" event={"ID":"6b796743-59cf-464d-90a7-b1b0226f9748","Type":"ContainerStarted","Data":"2d8aba4dada0f6d0e105923119ee96f364f09853ac85f72acbe47866c6715c85"} Nov 03 17:35:28.779232 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:28.779187 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podStartSLOduration=1.7791735279999998 podStartE2EDuration="1.779173528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:28.778562431 +0000 UTC m=+1071.233251478" watchObservedRunningTime="2026-11-03 17:35:28.779173528 +0000 UTC m=+1071.233862551" Nov 03 17:35:28.805091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:28.805013 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:32.268243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:32.266046 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:35:35.080102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:35.079694 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:35:35.080102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:35.079758 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:36.085211 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:36.085161 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:35:36.086138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:36.086107 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 03 17:35:37.805479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:37.805448 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:37.823222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:37.823192 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:37.839666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:37.839647 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:35:39.103138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.103050 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" containerID="cri-o://c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba" gracePeriod=45 Nov 03 17:35:39.363407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.363352 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:35:39.498086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498037 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498111 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498136 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498165 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498247 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\" (UniqueName: \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498334 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:39.498513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498347 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:39.498513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498433 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\") pod \"c3239d83-1741-44dc-95eb-2932236c2a02\" (UID: \"c3239d83-1741-44dc-95eb-2932236c2a02\") " Nov 03 17:35:39.498803 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.498782 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:39.499120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.499102 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.499199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.499128 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.499199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.499143 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.500930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.500902 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:39.501192 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.501166 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "kube-api-access-vmvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:39.501192 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.501174 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm" pod "c3239d83-1741-44dc-95eb-2932236c2a02" (UID: "c3239d83-1741-44dc-95eb-2932236c2a02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:39.599660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.599629 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3239d83-1741-44dc-95eb-2932236c2a02-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.599660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.599657 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3239d83-1741-44dc-95eb-2932236c2a02-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.599660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.599667 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmvc9\" (UniqueName: \"kubernetes.io/projected/c3239d83-1741-44dc-95eb-2932236c2a02-kube-api-access-vmvc9\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:35:39.828945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.828915 4885 generic.go:356] "Generic (PLEG): container finished" podID="c3239d83-1741-44dc-95eb-2932236c2a02" containerID="c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba" exitCode=0 Nov 03 17:35:39.829143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.828965 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" Nov 03 17:35:39.829143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.828996 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" event={"ID":"c3239d83-1741-44dc-95eb-2932236c2a02","Type":"ContainerDied","Data":"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba"} Nov 03 17:35:39.829143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.829034 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28" event={"ID":"c3239d83-1741-44dc-95eb-2932236c2a02","Type":"ContainerDied","Data":"6a2fa05b313f54f7031b0cf9dc32e0080c39d164e79240b614a0d9b103b05657"} Nov 03 17:35:39.829143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.829049 4885 scope.go:122] "RemoveContainer" containerID="c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba" Nov 03 17:35:39.838783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.838760 4885 scope.go:122] "RemoveContainer" containerID="c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba" Nov 03 17:35:39.839121 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:35:39.839102 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba\": container with ID starting with c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba not found: ID does not exist" containerID="c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba" Nov 03 17:35:39.839202 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.839132 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba"} err="failed to get container status \"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba\": rpc error: code = NotFound desc = could not find container \"c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba\": container with ID starting with c6359ccf9614a55a34e87799afb636bb1e9de7d6250960fdac1a302d1f3a3aba not found: ID does not exist" Nov 03 17:35:39.999846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:39.999769 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:35:40.007511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:40.007488 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/auth-test-model-147834-kserve-58fdcb9cdf-6lj28"] Nov 03 17:35:41.662872 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:35:41.662838 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" path="/var/lib/kubelet/pods/c3239d83-1741-44dc-95eb-2932236c2a02/volumes" Nov 03 17:35:48.605372 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:35:48.605335 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:36:00.997151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:00.994996 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:36:00.997151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:00.995688 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" Nov 03 17:36:00.997151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:00.995707 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" Nov 03 17:36:00.997151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:00.996440 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3239d83-1741-44dc-95eb-2932236c2a02" containerName="main" Nov 03 17:36:01.001802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.001780 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.071133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.071100 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e34dd2-openshift-default-dockercfg-66s84\"" Nov 03 17:36:01.165090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.125924 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:36:01.189359 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.135285 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfdb\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.189682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.189660 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.189882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.189865 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.190095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.190055 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.190385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.190367 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.190563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.190548 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.190723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.190707 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.190889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.190871 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.191117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.191099 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.295527 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.295438 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfdb\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.297059 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298199 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298247 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298294 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298329 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298374 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298410 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.298561 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.299228 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.300061 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.316090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.300409 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.317135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.317111 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.317891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.317868 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.356802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.330519 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.357034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.332366 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.376814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.376781 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfdb\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.448708 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.448675 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\") pod \"e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:01.646975 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:01.646939 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:02.083722 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:36:02.083688 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2193d231_ee89_4b18_9d25_12252d417008.slice/crio-05b8fcb172c898ffd4a1d71853120cdec392835095d76eb10a346bdd4a0254f7 WatchSource:0}: Error finding container 05b8fcb172c898ffd4a1d71853120cdec392835095d76eb10a346bdd4a0254f7: Status 404 returned error can't find the container with id 05b8fcb172c898ffd4a1d71853120cdec392835095d76eb10a346bdd4a0254f7 Nov 03 17:36:02.086021 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.085994 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:36:02.086160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.085989 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:36:02.086160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.086139 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:36:02.086276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.086212 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:36:02.962125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.962094 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" event={"ID":"2193d231-ee89-4b18-9d25-12252d417008","Type":"ContainerStarted","Data":"1c6d0455346de183c19bef72fc6b2286e9e58b5b6361f884267bdf6a0a0d6201"} Nov 03 17:36:02.962313 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:02.962150 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" event={"ID":"2193d231-ee89-4b18-9d25-12252d417008","Type":"ContainerStarted","Data":"05b8fcb172c898ffd4a1d71853120cdec392835095d76eb10a346bdd4a0254f7"} Nov 03 17:36:03.137736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:03.137672 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podStartSLOduration=3.137652981 podStartE2EDuration="3.137652981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:03.13682778 +0000 UTC m=+1105.591516803" watchObservedRunningTime="2026-11-03 17:36:03.137652981 +0000 UTC m=+1105.592342006" Nov 03 17:36:03.647859 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:03.647823 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:03.649582 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:03.649554 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 17:36:03.649714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:03.649601 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 17:36:04.647805 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:04.647771 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 17:36:04.648240 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:04.647825 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 17:36:05.647590 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:05.647554 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 17:36:05.647770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:05.647616 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 17:36:06.647612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:06.647566 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 17:36:06.648212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:06.647632 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 17:36:07.647807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:07.647766 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 17:36:07.648585 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:07.648551 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 17:36:08.652347 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:08.652315 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:08.652849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:08.652604 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:08.653390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:08.653374 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:36:18.604380 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:36:18.604342 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:36:21.834091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.832981 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:36:21.846719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.846687 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.893320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.893162 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.893843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.893804 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.894128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.894109 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.894369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.894351 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.894543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.894516 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.894675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.894661 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wwk\" (UniqueName: \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:21.900108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.900035 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:36:21.920061 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.920039 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-687fb3\"/\"distinct-2-w0-687fb3-kserve-self-signed-certs\"" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.999858 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.999965 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:21.999992 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.000050 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.000099 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.000127 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wwk\" (UniqueName: \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.004127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.000905 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.006020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.005986 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.016465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.016383 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.035276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.035212 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.038325 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.036690 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:36:22.038843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.038769 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.063655 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.063620 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wwk\" (UniqueName: \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\") pod \"distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.172175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.172138 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:22.837516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:22.837489 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:36:22.838779 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:36:22.838751 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e5f376_8ed7_49b4_8779_12218aa50387.slice/crio-00891ece491c692fd01e127cd960c12d96fdee7c40f0a632171a5a2355cae0ce WatchSource:0}: Error finding container 00891ece491c692fd01e127cd960c12d96fdee7c40f0a632171a5a2355cae0ce: Status 404 returned error can't find the container with id 00891ece491c692fd01e127cd960c12d96fdee7c40f0a632171a5a2355cae0ce Nov 03 17:36:23.117014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:23.116928 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" event={"ID":"58e5f376-8ed7-49b4-8779-12218aa50387","Type":"ContainerStarted","Data":"a52db638911aec2a4ce2ddaaff69682a101e743bae64b4fa62fe8d0e4fb45c93"} Nov 03 17:36:23.117014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:23.116970 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" event={"ID":"58e5f376-8ed7-49b4-8779-12218aa50387","Type":"ContainerStarted","Data":"00891ece491c692fd01e127cd960c12d96fdee7c40f0a632171a5a2355cae0ce"} Nov 03 17:36:23.175093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:23.174694 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:23.237953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:23.237799 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podStartSLOduration=2.237783414 podStartE2EDuration="2.237783414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:23.236848463 +0000 UTC m=+1125.691537488" watchObservedRunningTime="2026-11-03 17:36:23.237783414 +0000 UTC m=+1125.692472439" Nov 03 17:36:32.172466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:32.172435 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:32.187455 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:32.187432 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:33.182179 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:33.182148 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:36:37.838282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:37.838255 4885 scope.go:122] "RemoveContainer" containerID="6064b2f9a3eacc95e522a8c0591192da0300adeb08a41986956a4d2ff5d7b539" Nov 03 17:36:38.053358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:38.053324 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8"] Nov 03 17:36:38.103838 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:38.103751 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-739e8d90-wbdh8"] Nov 03 17:36:39.661531 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:39.661495 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f877f0-5ba2-4a8d-bb58-8286395b9dcb" path="/var/lib/kubelet/pods/a2f877f0-5ba2-4a8d-bb58-8286395b9dcb/volumes" Nov 03 17:36:48.605459 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:36:48.605425 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:36:53.182461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:36:53.182423 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:37:02.087882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.087848 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4"] Nov 03 17:37:02.092742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.092722 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.145820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.145787 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4"] Nov 03 17:37:02.165543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.165503 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4bk\" (UniqueName: \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.165730 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.165555 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.165730 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.165691 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.193624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.193598 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:37:02.194146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.194123 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:37:02.194259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.194169 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:37:02.197928 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.197910 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:37:02.267144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.267112 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.267338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.267222 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4bk\" (UniqueName: \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.267338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.267261 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.267749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.267727 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.269613 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.269592 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.335937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.335905 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4bk\" (UniqueName: \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\") pod \"maas-api-revoke-sub-960c2fb0-l62z4\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.405173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.405147 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:02.772050 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:37:02.772013 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7324d4_fff3_4ee8_9117_d318afdfe2c7.slice/crio-de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4 WatchSource:0}: Error finding container de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4: Status 404 returned error can't find the container with id de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4 Nov 03 17:37:02.774799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:02.774780 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4"] Nov 03 17:37:03.221007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:03.220975 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:37:03.402516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:03.401527 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" event={"ID":"ee7324d4-fff3-4ee8-9117-d318afdfe2c7","Type":"ContainerStarted","Data":"bb6ab4fb6db65a001de2d0f5346a006d8cb3bb633a44522bcf702258855558b9"} Nov 03 17:37:03.402516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:03.401594 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" event={"ID":"ee7324d4-fff3-4ee8-9117-d318afdfe2c7","Type":"ContainerStarted","Data":"de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4"} Nov 03 17:37:03.526263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:03.526213 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" podStartSLOduration=1.52619963 podStartE2EDuration="1.52619963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:03.524169249 +0000 UTC m=+1165.978858273" watchObservedRunningTime="2026-11-03 17:37:03.52619963 +0000 UTC m=+1165.980888664" Nov 03 17:37:04.408793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:04.408762 4885 generic.go:356] "Generic (PLEG): container finished" podID="ee7324d4-fff3-4ee8-9117-d318afdfe2c7" containerID="bb6ab4fb6db65a001de2d0f5346a006d8cb3bb633a44522bcf702258855558b9" exitCode=0 Nov 03 17:37:04.409307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:04.408845 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" event={"ID":"ee7324d4-fff3-4ee8-9117-d318afdfe2c7","Type":"ContainerDied","Data":"bb6ab4fb6db65a001de2d0f5346a006d8cb3bb633a44522bcf702258855558b9"} Nov 03 17:37:05.555927 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.555898 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:05.613160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.613135 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\") pod \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " Nov 03 17:37:05.613295 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.613202 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\") pod \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " Nov 03 17:37:05.613365 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.613343 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\" (UniqueName: \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\") pod \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\" (UID: \"ee7324d4-fff3-4ee8-9117-d318afdfe2c7\") " Nov 03 17:37:05.613560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.613534 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca" pod "ee7324d4-fff3-4ee8-9117-d318afdfe2c7" (UID: "ee7324d4-fff3-4ee8-9117-d318afdfe2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:05.613821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.613806 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:05.615336 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.615310 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca" pod "ee7324d4-fff3-4ee8-9117-d318afdfe2c7" (UID: "ee7324d4-fff3-4ee8-9117-d318afdfe2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:05.615484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.615462 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk" pod "ee7324d4-fff3-4ee8-9117-d318afdfe2c7" (UID: "ee7324d4-fff3-4ee8-9117-d318afdfe2c7"). InnerVolumeSpecName "kube-api-access-gz4bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:05.714936 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.714909 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:05.714936 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:05.714935 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4bk\" (UniqueName: \"kubernetes.io/projected/ee7324d4-fff3-4ee8-9117-d318afdfe2c7-kube-api-access-gz4bk\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:06.422181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:06.422150 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" Nov 03 17:37:06.422348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:06.422150 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4" event={"ID":"ee7324d4-fff3-4ee8-9117-d318afdfe2c7","Type":"ContainerDied","Data":"de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4"} Nov 03 17:37:06.422348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:06.422293 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6dc4eeab779f115f7f317b1670aebf596e8ecd575ea962cb0113a2f838c9e4" Nov 03 17:37:13.091215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:13.091161 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:13.114204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:13.114157 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:13.176216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:13.176161 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 17:37:13.176594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:13.176570 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:14.178644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:14.176629 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:14.178644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:14.177256 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:37:14.791889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:14.791764 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:37:17.561701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.561668 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm"] Nov 03 17:37:17.562150 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.562136 4885 cpu_manager.go:414] "containerMap: removing container" podUID="ee7324d4-fff3-4ee8-9117-d318afdfe2c7" containerName="revoke-keys" Nov 03 17:37:17.562150 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.562151 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7324d4-fff3-4ee8-9117-d318afdfe2c7" containerName="revoke-keys" Nov 03 17:37:17.562229 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.562221 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7324d4-fff3-4ee8-9117-d318afdfe2c7" containerName="revoke-keys" Nov 03 17:37:17.565333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.565307 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.575219 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.575197 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.575323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.575227 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j9rr\" (UniqueName: \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.575456 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.575438 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.621819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.621715 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:37:17.621819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.621791 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:37:17.673513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.673483 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:37:17.675113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.674831 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:37:17.676664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.676628 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.676786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.676690 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.676786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.676715 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j9rr\" (UniqueName: \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.677459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.677411 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm"] Nov 03 17:37:17.678201 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.678176 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.680455 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.680430 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.739842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.739810 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j9rr\" (UniqueName: \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\") pod \"maas-api-revoke-sub-fba755f0-stbfm\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:17.898686 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:17.888251 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:18.241397 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.241303 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" containerID="cri-o://309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20" gracePeriod=45 Nov 03 17:37:18.413156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.404008 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm"] Nov 03 17:37:18.614309 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:37:18.614269 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:37:18.646614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.646588 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:37:18.722172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722148 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722206 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\" (UniqueName: \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722236 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722288 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722320 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722350 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\" (UniqueName: \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\") pod \"12db0d7c-ef44-4fe5-a887-79740a5f3261\" (UID: \"12db0d7c-ef44-4fe5-a887-79740a5f3261\") " Nov 03 17:37:18.722576 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722552 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:18.722631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722606 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:18.722675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722631 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:18.722904 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722884 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.723005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722909 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.723005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.722925 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.724792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.724770 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:18.724792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.724777 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:18.724917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.724791 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx" pod "12db0d7c-ef44-4fe5-a887-79740a5f3261" (UID: "12db0d7c-ef44-4fe5-a887-79740a5f3261"). InnerVolumeSpecName "kube-api-access-c8mzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:18.753828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.753801 4885 generic.go:356] "Generic (PLEG): container finished" podID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerID="309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20" exitCode=0 Nov 03 17:37:18.753984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.753869 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" Nov 03 17:37:18.753984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.753890 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" event={"ID":"12db0d7c-ef44-4fe5-a887-79740a5f3261","Type":"ContainerDied","Data":"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20"} Nov 03 17:37:18.753984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.753934 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s" event={"ID":"12db0d7c-ef44-4fe5-a887-79740a5f3261","Type":"ContainerDied","Data":"7aa33591bd4c6e798632132e2569f82becda002b97e72aa563d7aed4c0a8cae1"} Nov 03 17:37:18.753984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.753959 4885 scope.go:122] "RemoveContainer" containerID="309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20" Nov 03 17:37:18.755331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.755310 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" event={"ID":"46205b2f-f17a-49ad-bfae-bbdb740c4a59","Type":"ContainerStarted","Data":"a6e23871a94f99842b11a5d6c62255baebc54808157c0c35acb416a6bfee0f4a"} Nov 03 17:37:18.764945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.764900 4885 scope.go:122] "RemoveContainer" containerID="309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20" Nov 03 17:37:18.765425 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:37:18.765393 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20\": container with ID starting with 309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20 not found: ID does not exist" containerID="309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20" Nov 03 17:37:18.765523 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.765436 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20"} err="failed to get container status \"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20\": rpc error: code = NotFound desc = could not find container \"309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20\": container with ID starting with 309cf0d6e068dae5e138592bf7107fee7a9a13a021453a3c2bbafc6b2c2a2f20 not found: ID does not exist" Nov 03 17:37:18.823608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.823537 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mzx\" (UniqueName: \"kubernetes.io/projected/12db0d7c-ef44-4fe5-a887-79740a5f3261-kube-api-access-c8mzx\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.823608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.823563 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12db0d7c-ef44-4fe5-a887-79740a5f3261-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.823608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.823575 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12db0d7c-ef44-4fe5-a887-79740a5f3261-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:18.883040 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.883008 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:37:18.886862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:18.886837 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e34dd2/test-model-e34dd2-kserve-76889756b7-v8l4s"] Nov 03 17:37:19.663004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:19.662974 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" path="/var/lib/kubelet/pods/12db0d7c-ef44-4fe5-a887-79740a5f3261/volumes" Nov 03 17:37:19.763280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:19.763248 4885 generic.go:356] "Generic (PLEG): container finished" podID="46205b2f-f17a-49ad-bfae-bbdb740c4a59" containerID="48d6531aeb35f84bc917c9aee468ad36cc989f3dc52e06b4d51b2c045f2fab71" exitCode=0 Nov 03 17:37:19.763439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:19.763332 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" event={"ID":"46205b2f-f17a-49ad-bfae-bbdb740c4a59","Type":"ContainerDied","Data":"48d6531aeb35f84bc917c9aee468ad36cc989f3dc52e06b4d51b2c045f2fab71"} Nov 03 17:37:20.912625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.912597 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:20.946331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.946299 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\") pod \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " Nov 03 17:37:20.946477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.946339 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\") pod \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " Nov 03 17:37:20.946522 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.946470 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\" (UniqueName: \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\") pod \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\" (UID: \"46205b2f-f17a-49ad-bfae-bbdb740c4a59\") " Nov 03 17:37:20.946681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.946659 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca" pod "46205b2f-f17a-49ad-bfae-bbdb740c4a59" (UID: "46205b2f-f17a-49ad-bfae-bbdb740c4a59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:20.946869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.946852 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:20.948535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.948516 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca" pod "46205b2f-f17a-49ad-bfae-bbdb740c4a59" (UID: "46205b2f-f17a-49ad-bfae-bbdb740c4a59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:20.948765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:20.948745 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr" pod "46205b2f-f17a-49ad-bfae-bbdb740c4a59" (UID: "46205b2f-f17a-49ad-bfae-bbdb740c4a59"). InnerVolumeSpecName "kube-api-access-8j9rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:21.048571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:21.048500 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46205b2f-f17a-49ad-bfae-bbdb740c4a59-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:21.048571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:21.048524 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j9rr\" (UniqueName: \"kubernetes.io/projected/46205b2f-f17a-49ad-bfae-bbdb740c4a59-kube-api-access-8j9rr\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:37:21.775511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:21.775484 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" Nov 03 17:37:21.775685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:21.775479 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm" event={"ID":"46205b2f-f17a-49ad-bfae-bbdb740c4a59","Type":"ContainerDied","Data":"a6e23871a94f99842b11a5d6c62255baebc54808157c0c35acb416a6bfee0f4a"} Nov 03 17:37:21.775685 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:21.775622 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e23871a94f99842b11a5d6c62255baebc54808157c0c35acb416a6bfee0f4a" Nov 03 17:37:30.350448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:30.350368 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:37:37.862220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:37.862192 4885 scope.go:122] "RemoveContainer" containerID="02827f4ffd2ac42226acd1d66a36acfd29c90a1a2e053744bee3b2cc6adc5ea1" Nov 03 17:37:48.606041 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:37:48.605999 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:37:52.001205 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:52.001160 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:52.119093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:52.108163 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:52.136127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:52.124514 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:52.817092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:52.816801 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:52.834214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:52.834167 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:53.004210 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:53.004157 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:53.005520 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:53.005495 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:37:53.185195 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:53.185152 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:37:53.239668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:53.239634 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:37:54.817424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:54.817389 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:37:54.817863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:37:54.817633 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" containerID="cri-o://d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795" gracePeriod=30 Nov 03 17:38:00.078351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.078327 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:38:00.089986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.089910 4885 generic.go:356] "Generic (PLEG): container finished" podID="63c55830-f20b-4846-a83f-1ad9d5249322" containerID="d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795" exitCode=0 Nov 03 17:38:00.089986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.089946 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" event={"ID":"63c55830-f20b-4846-a83f-1ad9d5249322","Type":"ContainerDied","Data":"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795"} Nov 03 17:38:00.089986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.089968 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" Nov 03 17:38:00.089986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.089987 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj" event={"ID":"63c55830-f20b-4846-a83f-1ad9d5249322","Type":"ContainerDied","Data":"ee67b5e42d1fbb9d5478e47bfb7ca0528a265334ba0300711fae9e3c12e0efa6"} Nov 03 17:38:00.090255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.090010 4885 scope.go:122] "RemoveContainer" containerID="d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795" Nov 03 17:38:00.101545 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.100304 4885 scope.go:122] "RemoveContainer" containerID="d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795" Nov 03 17:38:00.101701 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:38:00.101683 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795\": container with ID starting with d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795 not found: ID does not exist" containerID="d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795" Nov 03 17:38:00.101754 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.101714 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795"} err="failed to get container status \"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795\": rpc error: code = NotFound desc = could not find container \"d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795\": container with ID starting with d74e1a0ad590abbd82c1fbf4773918aaab0fbef7742bfdd1d59639721198a795 not found: ID does not exist" Nov 03 17:38:00.170178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170146 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170249 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170271 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170332 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170422 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170437 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:00.170470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170450 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170548 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170580 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170574 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:00.170765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170627 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\") pod \"63c55830-f20b-4846-a83f-1ad9d5249322\" (UID: \"63c55830-f20b-4846-a83f-1ad9d5249322\") " Nov 03 17:38:00.170765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170642 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:00.170765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170711 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:00.170765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.170735 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:00.171087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.171051 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.171087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.171089 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.171275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.171100 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.171275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.171109 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c55830-f20b-4846-a83f-1ad9d5249322-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.171275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.171120 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.172960 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.172935 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:00.173083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.173013 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "kube-api-access-xscjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:00.173240 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.173216 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:00.174183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.174156 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo" pod "63c55830-f20b-4846-a83f-1ad9d5249322" (UID: "63c55830-f20b-4846-a83f-1ad9d5249322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:38:00.271808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.271780 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.271808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.271807 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c55830-f20b-4846-a83f-1ad9d5249322-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.271808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.271817 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c55830-f20b-4846-a83f-1ad9d5249322-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.272046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.271831 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xscjh\" (UniqueName: \"kubernetes.io/projected/63c55830-f20b-4846-a83f-1ad9d5249322-kube-api-access-xscjh\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:00.544702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.539397 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:38:00.547338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:00.547302 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3f1cf1-openshift-default-f7964dddc-t2fcj"] Nov 03 17:38:01.661631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:01.661598 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" path="/var/lib/kubelet/pods/63c55830-f20b-4846-a83f-1ad9d5249322/volumes" Nov 03 17:38:07.033794 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:07.033761 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:38:09.313351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.313315 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 17:38:09.317738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.317706 4885 cpu_manager.go:414] "containerMap: removing container" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" Nov 03 17:38:09.317932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.317915 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" Nov 03 17:38:09.318090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318048 4885 cpu_manager.go:414] "containerMap: removing container" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" Nov 03 17:38:09.318196 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318181 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" Nov 03 17:38:09.318299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318288 4885 cpu_manager.go:414] "containerMap: removing container" podUID="46205b2f-f17a-49ad-bfae-bbdb740c4a59" containerName="revoke-keys" Nov 03 17:38:09.318379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318368 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46205b2f-f17a-49ad-bfae-bbdb740c4a59" containerName="revoke-keys" Nov 03 17:38:09.318595 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318580 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="12db0d7c-ef44-4fe5-a887-79740a5f3261" containerName="main" Nov 03 17:38:09.318707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318696 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c55830-f20b-4846-a83f-1ad9d5249322" containerName="istio-proxy" Nov 03 17:38:09.318795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.318785 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="46205b2f-f17a-49ad-bfae-bbdb740c4a59" containerName="revoke-keys" Nov 03 17:38:09.330093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.323484 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.385090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.378652 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-8c9b0f-openshift-default-dockercfg-5s7kz\"" Nov 03 17:38:09.439145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.439110 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 17:38:09.445282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.445248 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.464288 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.464243 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.464965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.464938 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzcz\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.469252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.469220 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.469420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.469403 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.469624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.469608 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.469781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.469765 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.469953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.469925 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.492814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.492774 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.622964 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.623083 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.623118 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzcz\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.623177 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.623208 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.630156 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.623282 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.653093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.652250 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.653093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.652316 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.653093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.652380 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.653093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.652846 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.699012 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.698964 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.734096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.706054 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.734457 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.711496 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.734600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.711880 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.739736 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.739686 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.755438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.755015 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:09.819094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.794382 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 17:38:09.819094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.795252 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.850099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.850036 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.876749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.876717 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzcz\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:09.984395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:09.984301 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:10.241606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:10.241164 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:10.804094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:10.798208 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:10.831175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:10.817690 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:38:10.831490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:10.831454 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:11.067189 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.067154 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 17:38:11.099277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.099183 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 03 17:38:11.149424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.149351 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:11.307099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.288175 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:11.334110 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.317127 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:38:11.419127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.412220 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:38:11.428019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.419321 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:38:11.428019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.426933 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:38:11.432182 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.432143 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:38:11.450678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.450657 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:38:11.477173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:11.477143 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" event={"ID":"9936961a-214f-4d66-b54f-6f2ce6b15376","Type":"ContainerStarted","Data":"309c27e17ed404d0027a28ff79ab7a746b9372f012f7be376d497bf01eb08997"} Nov 03 17:38:12.484550 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:12.484513 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" event={"ID":"9936961a-214f-4d66-b54f-6f2ce6b15376","Type":"ContainerStarted","Data":"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df"} Nov 03 17:38:12.671243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:12.671187 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podStartSLOduration=3.671172303 podStartE2EDuration="3.671172303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:12.659563411 +0000 UTC m=+1235.114252464" watchObservedRunningTime="2026-11-03 17:38:12.671172303 +0000 UTC m=+1235.125861326" Nov 03 17:38:12.985155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:12.985122 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:12.986759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:12.986736 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:38:12.986885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:12.986779 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 17:38:13.986186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:13.986151 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:38:13.986580 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:13.986208 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 17:38:14.986178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:14.986137 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:38:14.986388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:14.986192 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 17:38:15.289142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.289039 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5"] Nov 03 17:38:15.292850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.292832 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.347200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.347170 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5"] Nov 03 17:38:15.376765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.376737 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.376931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.376778 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpmd\" (UniqueName: \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.376931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.376911 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.395831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.395804 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:38:15.395831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.395813 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:38:15.395994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.395840 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:38:15.398556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.398543 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:38:15.478142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.478110 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.478327 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.478166 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpmd\" (UniqueName: \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.478327 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.478283 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.479035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.479009 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.480743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.480719 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.541315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.541237 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpmd\" (UniqueName: \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\") pod \"maas-api-revoke-sub-93a9d869-w2vg5\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.625091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.624239 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:15.985698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.985647 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:38:15.985861 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:15.985714 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 17:38:16.035341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.035304 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5"] Nov 03 17:38:16.532882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.532773 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" event={"ID":"59222f9e-f097-4ad3-bafb-54d2afd7a7fa","Type":"ContainerStarted","Data":"be327d78f01c2f850848330414b6774e875fbcb4676065c3f1af626aca64eed3"} Nov 03 17:38:16.532882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.532846 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" event={"ID":"59222f9e-f097-4ad3-bafb-54d2afd7a7fa","Type":"ContainerStarted","Data":"d8e31a9443eac9254fed1f94842d3a157105af3091db917e7c771d0afd2bfbb6"} Nov 03 17:38:16.653266 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.653201 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" podStartSLOduration=1.653179833 podStartE2EDuration="1.653179833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:16.652513399 +0000 UTC m=+1239.107202423" watchObservedRunningTime="2026-11-03 17:38:16.653179833 +0000 UTC m=+1239.107868858" Nov 03 17:38:16.985539 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.985498 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:38:16.985741 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:16.985584 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 17:38:17.541434 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:17.541395 4885 generic.go:356] "Generic (PLEG): container finished" podID="59222f9e-f097-4ad3-bafb-54d2afd7a7fa" containerID="be327d78f01c2f850848330414b6774e875fbcb4676065c3f1af626aca64eed3" exitCode=0 Nov 03 17:38:17.541919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:17.541483 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" event={"ID":"59222f9e-f097-4ad3-bafb-54d2afd7a7fa","Type":"ContainerDied","Data":"be327d78f01c2f850848330414b6774e875fbcb4676065c3f1af626aca64eed3"} Nov 03 17:38:17.989587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:17.989557 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:17.990084 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:17.990054 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:17.990651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:17.990634 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 17:38:18.623469 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:38:18.623398 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:38:18.693488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.693468 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:18.814252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.814160 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\") pod \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " Nov 03 17:38:18.814252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.814228 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\") pod \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " Nov 03 17:38:18.814488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.814431 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\" (UniqueName: \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\") pod \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\" (UID: \"59222f9e-f097-4ad3-bafb-54d2afd7a7fa\") " Nov 03 17:38:18.814588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.814563 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca" pod "59222f9e-f097-4ad3-bafb-54d2afd7a7fa" (UID: "59222f9e-f097-4ad3-bafb-54d2afd7a7fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:18.814821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.814800 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:18.816572 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.816541 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca" pod "59222f9e-f097-4ad3-bafb-54d2afd7a7fa" (UID: "59222f9e-f097-4ad3-bafb-54d2afd7a7fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:18.816721 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.816699 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd" pod "59222f9e-f097-4ad3-bafb-54d2afd7a7fa" (UID: "59222f9e-f097-4ad3-bafb-54d2afd7a7fa"). InnerVolumeSpecName "kube-api-access-djpmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:18.916405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.916367 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpmd\" (UniqueName: \"kubernetes.io/projected/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-kube-api-access-djpmd\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:18.916405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:18.916396 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59222f9e-f097-4ad3-bafb-54d2afd7a7fa-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:19.050208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.050170 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs"] Nov 03 17:38:19.052700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.052675 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ef76450-bhvgs"] Nov 03 17:38:19.553841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.553803 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" event={"ID":"59222f9e-f097-4ad3-bafb-54d2afd7a7fa","Type":"ContainerDied","Data":"d8e31a9443eac9254fed1f94842d3a157105af3091db917e7c771d0afd2bfbb6"} Nov 03 17:38:19.553841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.553842 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5" Nov 03 17:38:19.554057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.553883 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e31a9443eac9254fed1f94842d3a157105af3091db917e7c771d0afd2bfbb6" Nov 03 17:38:19.663170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:19.663132 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbfcfa1-8abb-40e5-912f-dd66fe607d91" path="/var/lib/kubelet/pods/efbfcfa1-8abb-40e5-912f-dd66fe607d91/volumes" Nov 03 17:38:28.668896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.668866 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w"] Nov 03 17:38:28.669333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.669321 4885 cpu_manager.go:414] "containerMap: removing container" podUID="59222f9e-f097-4ad3-bafb-54d2afd7a7fa" containerName="revoke-keys" Nov 03 17:38:28.669381 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.669334 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59222f9e-f097-4ad3-bafb-54d2afd7a7fa" containerName="revoke-keys" Nov 03 17:38:28.669413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.669407 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="59222f9e-f097-4ad3-bafb-54d2afd7a7fa" containerName="revoke-keys" Nov 03 17:38:28.672972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.672945 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.745706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.745673 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:38:28.786383 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.786345 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:38:28.786383 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.786346 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:38:28.786613 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.786477 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:38:28.789986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.789963 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w"] Nov 03 17:38:28.797057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.797031 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.797279 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.797261 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqc5\" (UniqueName: \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.797398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.797308 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.898386 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.898351 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqc5\" (UniqueName: \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.898606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.898413 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.898606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.898456 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.899262 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.899239 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.901420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.901396 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.960734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.960664 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqc5\" (UniqueName: \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\") pod \"maas-api-revoke-sub-5734ce0c-d6r8w\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:28.986234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:28.986207 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:29.380371 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:38:29.380317 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75172a92_cf86_4a50_97d0_f3ad832b7c46.slice/crio-e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71 WatchSource:0}: Error finding container e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71: Status 404 returned error can't find the container with id e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71 Nov 03 17:38:29.383135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:29.383112 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w"] Nov 03 17:38:29.673817 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:29.673745 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" event={"ID":"75172a92-cf86-4a50-97d0-f3ad832b7c46","Type":"ContainerStarted","Data":"e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71"} Nov 03 17:38:30.681951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:30.681915 4885 generic.go:356] "Generic (PLEG): container finished" podID="75172a92-cf86-4a50-97d0-f3ad832b7c46" containerID="e3622186581088832e104e11be5d8932dbe8103bb055c45cf73243d4dbc78cf1" exitCode=0 Nov 03 17:38:30.682419 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:30.682004 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" event={"ID":"75172a92-cf86-4a50-97d0-f3ad832b7c46","Type":"ContainerDied","Data":"e3622186581088832e104e11be5d8932dbe8103bb055c45cf73243d4dbc78cf1"} Nov 03 17:38:31.828994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.828963 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:31.871500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.871472 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\" (UniqueName: \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\") pod \"75172a92-cf86-4a50-97d0-f3ad832b7c46\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " Nov 03 17:38:31.871679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.871561 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\") pod \"75172a92-cf86-4a50-97d0-f3ad832b7c46\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " Nov 03 17:38:31.871679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.871593 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\") pod \"75172a92-cf86-4a50-97d0-f3ad832b7c46\" (UID: \"75172a92-cf86-4a50-97d0-f3ad832b7c46\") " Nov 03 17:38:31.871917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.871890 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca" pod "75172a92-cf86-4a50-97d0-f3ad832b7c46" (UID: "75172a92-cf86-4a50-97d0-f3ad832b7c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:31.872095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.871982 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:31.873889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.873865 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca" pod "75172a92-cf86-4a50-97d0-f3ad832b7c46" (UID: "75172a92-cf86-4a50-97d0-f3ad832b7c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:31.873889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.873873 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5" pod "75172a92-cf86-4a50-97d0-f3ad832b7c46" (UID: "75172a92-cf86-4a50-97d0-f3ad832b7c46"). InnerVolumeSpecName "kube-api-access-dvqc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:31.973315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.973242 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75172a92-cf86-4a50-97d0-f3ad832b7c46-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:31.973315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:31.973270 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqc5\" (UniqueName: \"kubernetes.io/projected/75172a92-cf86-4a50-97d0-f3ad832b7c46-kube-api-access-dvqc5\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:32.693640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:32.693609 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" Nov 03 17:38:32.693819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:32.693639 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w" event={"ID":"75172a92-cf86-4a50-97d0-f3ad832b7c46","Type":"ContainerDied","Data":"e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71"} Nov 03 17:38:32.693819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:32.693687 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8745a4d580091939784fc37d86d18904b8ef702a5c3ff7f0d6cf826d6038b71" Nov 03 17:38:35.800007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:35.799976 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:38:35.800408 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:35.800224 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" containerID="cri-o://1c6d0455346de183c19bef72fc6b2286e9e58b5b6361f884267bdf6a0a0d6201" gracePeriod=30 Nov 03 17:38:37.257523 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:37.257494 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:38:38.671268 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:38.671232 4885 patch_prober.go:29] interesting pod/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:38:38.671894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:38.671858 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:38:41.085257 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.085201 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 03 17:38:41.797843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.797802 4885 generic.go:356] "Generic (PLEG): container finished" podID="2193d231-ee89-4b18-9d25-12252d417008" containerID="1c6d0455346de183c19bef72fc6b2286e9e58b5b6361f884267bdf6a0a0d6201" exitCode=0 Nov 03 17:38:41.797985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.797855 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" event={"ID":"2193d231-ee89-4b18-9d25-12252d417008","Type":"ContainerDied","Data":"1c6d0455346de183c19bef72fc6b2286e9e58b5b6361f884267bdf6a0a0d6201"} Nov 03 17:38:41.829647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.829385 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:38:41.899494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899461 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.899701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899522 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.899701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899572 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.899701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899615 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.899701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899695 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.900027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899761 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.900027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899792 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.900027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899825 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.900027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899867 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\") pod \"2193d231-ee89-4b18-9d25-12252d417008\" (UID: \"2193d231-ee89-4b18-9d25-12252d417008\") " Nov 03 17:38:41.900027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.899885 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:41.900358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.900338 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:41.900358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.900347 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:41.900665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.900468 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:41.900665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.900515 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:41.900665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.900599 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:41.902672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.902645 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:41.902793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.902724 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:38:41.904138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.904115 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:41.904629 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:41.904605 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb" pod "2193d231-ee89-4b18-9d25-12252d417008" (UID: "2193d231-ee89-4b18-9d25-12252d417008"). InnerVolumeSpecName "kube-api-access-hcfdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001395 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfdb\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-kube-api-access-hcfdb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001421 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2193d231-ee89-4b18-9d25-12252d417008-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001432 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001440 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001450 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2193d231-ee89-4b18-9d25-12252d417008-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001457 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2193d231-ee89-4b18-9d25-12252d417008-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001467 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.001471 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.001475 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2193d231-ee89-4b18-9d25-12252d417008-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:42.805121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.805060 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" Nov 03 17:38:42.805121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.805096 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b" event={"ID":"2193d231-ee89-4b18-9d25-12252d417008","Type":"ContainerDied","Data":"05b8fcb172c898ffd4a1d71853120cdec392835095d76eb10a346bdd4a0254f7"} Nov 03 17:38:42.805650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.805170 4885 scope.go:122] "RemoveContainer" containerID="1c6d0455346de183c19bef72fc6b2286e9e58b5b6361f884267bdf6a0a0d6201" Nov 03 17:38:42.986978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.986946 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:38:42.993788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:42.993762 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e34dd2-openshift-default-7cbdf86cbf-kvv9b"] Nov 03 17:38:43.661901 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:43.661867 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2193d231-ee89-4b18-9d25-12252d417008" path="/var/lib/kubelet/pods/2193d231-ee89-4b18-9d25-12252d417008/volumes" Nov 03 17:38:44.888305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888276 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9"] Nov 03 17:38:44.888733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888720 4885 cpu_manager.go:414] "containerMap: removing container" podUID="75172a92-cf86-4a50-97d0-f3ad832b7c46" containerName="revoke-keys" Nov 03 17:38:44.888775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888735 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75172a92-cf86-4a50-97d0-f3ad832b7c46" containerName="revoke-keys" Nov 03 17:38:44.888775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888746 4885 cpu_manager.go:414] "containerMap: removing container" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" Nov 03 17:38:44.888775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888751 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" Nov 03 17:38:44.888866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888832 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="75172a92-cf86-4a50-97d0-f3ad832b7c46" containerName="revoke-keys" Nov 03 17:38:44.888866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.888839 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2193d231-ee89-4b18-9d25-12252d417008" containerName="istio-proxy" Nov 03 17:38:44.894261 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.894232 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:44.932084 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.932038 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kzb\" (UniqueName: \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:44.932234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.932129 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:44.932234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.932158 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:44.946114 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.946085 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9"] Nov 03 17:38:44.994656 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.994618 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:38:44.994816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.994682 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:38:44.994816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.994750 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:38:44.999437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:44.999423 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:38:45.033213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.033183 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kzb\" (UniqueName: \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.033360 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.033224 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.033360 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.033240 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.033831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.033814 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.035691 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.035676 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.097770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.097737 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kzb\" (UniqueName: \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\") pod \"maas-api-revoke-sub-0a401a1c-l9lt9\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.207845 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.207765 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:45.573571 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:38:45.573544 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0d1cfe_c14f_4779_9375_a8636cae4e02.slice/crio-bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639 WatchSource:0}: Error finding container bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639: Status 404 returned error can't find the container with id bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639 Nov 03 17:38:45.576913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.576662 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9"] Nov 03 17:38:45.823929 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:45.823854 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" event={"ID":"bb0d1cfe-c14f-4779-9375-a8636cae4e02","Type":"ContainerStarted","Data":"bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639"} Nov 03 17:38:46.831020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:46.830931 4885 generic.go:356] "Generic (PLEG): container finished" podID="bb0d1cfe-c14f-4779-9375-a8636cae4e02" containerID="c01b812936876d3a0238f94d47a271e994836c77627ca7d742efb14e0c8d7506" exitCode=0 Nov 03 17:38:46.831020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:46.830984 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" event={"ID":"bb0d1cfe-c14f-4779-9375-a8636cae4e02","Type":"ContainerDied","Data":"c01b812936876d3a0238f94d47a271e994836c77627ca7d742efb14e0c8d7506"} Nov 03 17:38:47.973571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:47.973548 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:48.119406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.119380 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\") pod \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " Nov 03 17:38:48.119580 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.119484 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\" (UniqueName: \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\") pod \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " Nov 03 17:38:48.119640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.119626 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\") pod \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\" (UID: \"bb0d1cfe-c14f-4779-9375-a8636cae4e02\") " Nov 03 17:38:48.120027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.119996 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca" pod "bb0d1cfe-c14f-4779-9375-a8636cae4e02" (UID: "bb0d1cfe-c14f-4779-9375-a8636cae4e02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:48.121798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.121775 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb" pod "bb0d1cfe-c14f-4779-9375-a8636cae4e02" (UID: "bb0d1cfe-c14f-4779-9375-a8636cae4e02"). InnerVolumeSpecName "kube-api-access-f7kzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:48.122199 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.122181 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca" pod "bb0d1cfe-c14f-4779-9375-a8636cae4e02" (UID: "bb0d1cfe-c14f-4779-9375-a8636cae4e02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:48.221600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.221568 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kzb\" (UniqueName: \"kubernetes.io/projected/bb0d1cfe-c14f-4779-9375-a8636cae4e02-kube-api-access-f7kzb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:48.221600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.221593 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:48.221600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.221604 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0d1cfe-c14f-4779-9375-a8636cae4e02-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:38:48.606610 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:38:48.606524 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:38:48.842639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.842612 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" Nov 03 17:38:48.842803 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.842608 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9" event={"ID":"bb0d1cfe-c14f-4779-9375-a8636cae4e02","Type":"ContainerDied","Data":"bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639"} Nov 03 17:38:48.842803 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:48.842732 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd771c8b58e04a6b625eae687ece23c8e182fc5033ece73037490d8bc9c57639" Nov 03 17:38:54.989574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:54.989538 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:38:58.125232 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:58.125183 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:58.472145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:58.463642 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:58.818152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:58.812324 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 03 17:38:58.835091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:58.831156 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:38:59.103661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:38:59.103444 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:14.087228 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:14.087148 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:15.135221 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:15.135169 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:15.137705 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:15.137670 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:18.605832 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:39:18.605795 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:39:26.929843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.929803 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js"] Nov 03 17:39:26.932149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.930261 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bb0d1cfe-c14f-4779-9375-a8636cae4e02" containerName="revoke-keys" Nov 03 17:39:26.932149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.930273 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0d1cfe-c14f-4779-9375-a8636cae4e02" containerName="revoke-keys" Nov 03 17:39:26.932149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.930358 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0d1cfe-c14f-4779-9375-a8636cae4e02" containerName="revoke-keys" Nov 03 17:39:26.933218 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.933203 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:26.988198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:26.988166 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js"] Nov 03 17:39:27.007253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.007221 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.007416 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.007259 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.007416 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.007295 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt858\" (UniqueName: \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.035914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.035883 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:39:27.035914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.035898 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:39:27.036341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.036325 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:39:27.038839 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.038819 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:39:27.126083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.121842 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.126083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.121898 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.126083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.121942 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt858\" (UniqueName: \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.139824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.139793 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.140214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.140181 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.185491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.185422 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt858\" (UniqueName: \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\") pod \"maas-api-revoke-sub-6f41ec4e-2x7js\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.245740 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.245702 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:27.621814 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:39:27.621780 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad0c6f4_4f4f_40bc_ad13_09f22aac4392.slice/crio-2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212 WatchSource:0}: Error finding container 2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212: Status 404 returned error can't find the container with id 2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212 Nov 03 17:39:27.623440 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:27.623417 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js"] Nov 03 17:39:28.211724 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:28.211669 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" event={"ID":"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392","Type":"ContainerStarted","Data":"2d17dea77e288423cf2d3cbfce78021c6f62321dd71a6876cc8a301d0be15902"} Nov 03 17:39:28.212237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:28.211735 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" event={"ID":"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392","Type":"ContainerStarted","Data":"2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212"} Nov 03 17:39:28.336611 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:28.336532 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" podStartSLOduration=2.336511322 podStartE2EDuration="2.336511322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:28.334151508 +0000 UTC m=+1310.788840555" watchObservedRunningTime="2026-11-03 17:39:28.336511322 +0000 UTC m=+1310.791200356" Nov 03 17:39:29.219601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:29.219566 4885 generic.go:356] "Generic (PLEG): container finished" podID="2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" containerID="2d17dea77e288423cf2d3cbfce78021c6f62321dd71a6876cc8a301d0be15902" exitCode=0 Nov 03 17:39:29.220020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:29.219657 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" event={"ID":"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392","Type":"ContainerDied","Data":"2d17dea77e288423cf2d3cbfce78021c6f62321dd71a6876cc8a301d0be15902"} Nov 03 17:39:29.681767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:29.681733 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:39:30.061460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.061379 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx"] Nov 03 17:39:30.067722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.067694 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4974004a-r89sx"] Nov 03 17:39:30.476940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.476323 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:30.638138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.633495 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\") pod \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " Nov 03 17:39:30.638138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.633560 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\") pod \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " Nov 03 17:39:30.638138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.633691 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\" (UniqueName: \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\") pod \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\" (UID: \"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392\") " Nov 03 17:39:30.682093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.661301 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca" pod "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" (UID: "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:39:30.703100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.690941 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858" pod "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" (UID: "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392"). InnerVolumeSpecName "kube-api-access-gt858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:30.703100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.695190 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca" pod "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" (UID: "2ad0c6f4-4f4f-40bc-ad13-09f22aac4392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:30.792092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.791336 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt858\" (UniqueName: \"kubernetes.io/projected/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-kube-api-access-gt858\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:39:30.792092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.791374 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:39:30.792092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:30.791391 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:39:31.284093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:31.283302 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" event={"ID":"2ad0c6f4-4f4f-40bc-ad13-09f22aac4392","Type":"ContainerDied","Data":"2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212"} Nov 03 17:39:31.284093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:31.283369 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbe85301d74bc7e622eb7099d46dd6b329fa3dc7ba9864adead4fc3a6350212" Nov 03 17:39:31.284093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:31.283546 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js" Nov 03 17:39:31.750096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:31.749756 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3002c784-a018-4389-bba1-e76fe9f21275" path="/var/lib/kubelet/pods/3002c784-a018-4389-bba1-e76fe9f21275/volumes" Nov 03 17:39:33.459097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:33.458337 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 17:39:34.085206 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:34.085157 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:34.107250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:34.107204 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:34.402185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:34.401798 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:34.402185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:34.401860 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:37.923036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:37.923005 4885 scope.go:122] "RemoveContainer" containerID="dd1dd7fe780776555a6f4d79821528de5fa83304c4f2a6c7a97a294998f37e79" Nov 03 17:39:48.607920 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:39:48.607877 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:39:49.052324 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:49.052290 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv"] Nov 03 17:39:49.103351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:49.103311 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c73f7358-8ncjv"] Nov 03 17:39:49.664553 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:39:49.664514 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be" path="/var/lib/kubelet/pods/a3a5eaba-b4e7-4e75-b6b7-fc053cd6d5be/volumes" Nov 03 17:40:04.056442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:04.056409 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq"] Nov 03 17:40:04.107512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:04.107485 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aa167c-x5wdq"] Nov 03 17:40:05.662091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:05.662042 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee6a1c8-420f-4466-9c98-a3b151317b40" path="/var/lib/kubelet/pods/4ee6a1c8-420f-4466-9c98-a3b151317b40/volumes" Nov 03 17:40:11.044011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:11.043965 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:40:18.605940 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:40:18.605903 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:40:37.948808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:37.948734 4885 scope.go:122] "RemoveContainer" containerID="5cef0f7cead88bf431bc022345c9b7d2e721c4892922b8f57a944925c84a4877" Nov 03 17:40:41.015554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:40:41.015518 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:40:48.609934 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:40:48.609890 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:41:00.828663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:00.828614 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:01.090155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:01.089201 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:01.182312 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:01.182253 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:01.435236 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:01.435188 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:01.457187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:01.457132 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:16.658215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.658175 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:41:16.686134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.686098 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:41:16.686547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.686534 4885 cpu_manager.go:414] "containerMap: removing container" podUID="2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" containerName="revoke-keys" Nov 03 17:41:16.686591 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.686548 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" containerName="revoke-keys" Nov 03 17:41:16.686625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.686608 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" containerName="revoke-keys" Nov 03 17:41:16.689788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.689772 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.730970 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.730931 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.731165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.730983 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.731165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.731105 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.731165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.731154 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfhp\" (UniqueName: \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.731388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.731368 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.731474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.731458 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.744162 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.744135 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-49c123d1\"/\"kube-root-ca.crt\"" Nov 03 17:41:16.744335 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.744297 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-49c123d1\"/\"default-dockercfg-4hcs2\"" Nov 03 17:41:16.794766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.794731 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-49c123d1\"/\"openshift-service-ca.crt\"" Nov 03 17:41:16.794766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.794754 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-49c123d1\"/\"e2e-lifecycle-model-49c123d1-kserve-self-signed-certs\"" Nov 03 17:41:16.797405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.797378 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:41:16.854135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.840368 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.854463 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.854432 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.854662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.854643 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.854838 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.854814 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfhp\" (UniqueName: \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.855086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.855053 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.867286 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.867248 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.868055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.857475 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.868268 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.844532 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.868468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.867838 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.898693 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.898659 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.899391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.899355 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:16.919441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:16.919366 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfhp\" (UniqueName: \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\") pod \"e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:17.001527 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:17.001470 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:17.388311 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:41:17.388286 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1dbf390_a775_4a9c_8501_af0ec64ff19d.slice/crio-5b533d914cfccd5c5cb6303dbe93f70afd288961d196e8824afc5bd17991ed91 WatchSource:0}: Error finding container 5b533d914cfccd5c5cb6303dbe93f70afd288961d196e8824afc5bd17991ed91: Status 404 returned error can't find the container with id 5b533d914cfccd5c5cb6303dbe93f70afd288961d196e8824afc5bd17991ed91 Nov 03 17:41:17.390954 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:17.390933 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:41:18.246757 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:18.246710 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" event={"ID":"f1dbf390-a775-4a9c-8501-af0ec64ff19d","Type":"ContainerStarted","Data":"389f760274f985988a6599d18ddf8ed28bbc6cda77d63a8beaebbef36d07cd5d"} Nov 03 17:41:18.247359 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:18.246773 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" event={"ID":"f1dbf390-a775-4a9c-8501-af0ec64ff19d","Type":"ContainerStarted","Data":"5b533d914cfccd5c5cb6303dbe93f70afd288961d196e8824afc5bd17991ed91"} Nov 03 17:41:18.367889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:18.367825 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" podStartSLOduration=2.367805448 podStartE2EDuration="2.367805448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:18.367462591 +0000 UTC m=+1420.822151618" watchObservedRunningTime="2026-11-03 17:41:18.367805448 +0000 UTC m=+1420.822494475" Nov 03 17:41:18.614138 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:41:18.613977 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:41:19.003092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:19.002150 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:25.365358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.365318 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:41:25.369242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.369218 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.425959 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.425918 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:41:25.426599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.426570 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sht\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.426742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.426721 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.426924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.426886 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.427007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.426966 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.473196 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.473166 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 17:41:25.473196 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.473186 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 17:41:25.473396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.473186 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-03ea5a1192b1\"" Nov 03 17:41:25.473396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.473166 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 17:41:25.474575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.474562 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 17:41:25.474617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.474578 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2j857\"" Nov 03 17:41:25.528053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528021 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.528249 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528062 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sht\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.528249 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528182 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.528343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528248 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.528743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528711 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.528908 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.528887 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.530690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.530671 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.591104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.591048 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sht\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\") pod \"payload-processing-external-model-67bfdd8c49-4rgpl\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:25.683264 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:25.683235 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:26.054323 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:41:26.054288 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ff6d4d_cf62_4f0f_bc38_dc6c6988f7df.slice/crio-0ab96eecf6446be21c20ae5dcee3b10aade31d2fdad90a736dbe10471a0919e6 WatchSource:0}: Error finding container 0ab96eecf6446be21c20ae5dcee3b10aade31d2fdad90a736dbe10471a0919e6: Status 404 returned error can't find the container with id 0ab96eecf6446be21c20ae5dcee3b10aade31d2fdad90a736dbe10471a0919e6 Nov 03 17:41:26.055573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:26.055551 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:41:26.056855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:26.056834 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:41:26.312377 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:26.312295 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" event={"ID":"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df","Type":"ContainerStarted","Data":"0ab96eecf6446be21c20ae5dcee3b10aade31d2fdad90a736dbe10471a0919e6"} Nov 03 17:41:27.002791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:27.002758 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:27.015644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:27.015618 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:27.336315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:27.336240 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:41:29.334815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:29.334779 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" event={"ID":"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df","Type":"ContainerStarted","Data":"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52"} Nov 03 17:41:29.335248 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:29.334961 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:41:29.455194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:29.455141 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podStartSLOduration=1.6629448610000002 podStartE2EDuration="4.455127739s" totalImagesPullingTime="2.792182878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:29.45402235 +0000 UTC m=+1431.908711387" watchObservedRunningTime="2026-11-03 17:41:29.455127739 +0000 UTC m=+1431.909816764" Nov 03 17:41:33.752090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.745045 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:41:33.784095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.782366 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.808445 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.799267 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:41:33.818364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.818342 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.853678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.853640 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.853976 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.853957 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.854173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.854155 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.854834 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.854813 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w725\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.855033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855017 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.855308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855290 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.855480 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855466 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.855641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855626 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.855820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855805 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.855993 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.855978 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.856172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.856156 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.856405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.856388 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74xk\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.856602 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.856584 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.857205 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.857186 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.857396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.857380 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.857554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.857540 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.857704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.857691 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.857867 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.857852 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.866279 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.865645 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:41:33.902891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.902861 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:41:33.973094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.961680 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74xk\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.973362 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973337 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.973517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973502 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.973628 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973613 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.973730 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973717 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.973846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973832 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.983093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.973941 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.983470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.983445 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.983622 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.983608 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.983757 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.983743 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.983928 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.983908 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w725\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.984093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984057 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.984244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984229 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.984386 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984370 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.984561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984531 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.984783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984768 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.984941 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984926 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.985121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.985105 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.987671 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.986934 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.987671 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.987206 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.988243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.988212 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.988447 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.982965 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.988555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984845 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.988555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.984637 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.988701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.988418 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.988863 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.988728 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.989048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.988935 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.989176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.982520 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.989287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.989178 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:33.993767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.993723 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.994907 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.994881 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:33.996559 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:33.996539 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:34.064574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.064161 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:34.066055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.066022 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w725\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\") pod \"maas-default-gateway-openshift-default-58c6585447-n5wkm\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:34.066551 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.066523 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:34.067774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.067739 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74xk\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\") pod \"maas-default-gateway-openshift-default-58c6585447-c6nsc\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:34.124297 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.124262 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:34.175100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.174846 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:34.759142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.758370 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:41:34.887821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.887779 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:34.888078 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.888042 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:34.888222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:34.888205 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:35.141105 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.134992 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:41:35.168573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.167918 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:35.168573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.168014 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:35.168573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.168059 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:41:35.400687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.400635 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" event={"ID":"98023514-7416-4f9b-a29d-cff2720447d3","Type":"ContainerStarted","Data":"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609"} Nov 03 17:41:35.400895 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.400710 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" event={"ID":"98023514-7416-4f9b-a29d-cff2720447d3","Type":"ContainerStarted","Data":"45842210271c418d012fb24600954bf0cd023ae25dd19dae36c26148844d8e69"} Nov 03 17:41:35.484569 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.482999 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" event={"ID":"76a06ce6-651d-461f-8631-c9dfedce328b","Type":"ContainerStarted","Data":"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7"} Nov 03 17:41:35.484569 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.483122 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" event={"ID":"76a06ce6-651d-461f-8631-c9dfedce328b","Type":"ContainerStarted","Data":"a5a1543f85dbf8d6a8aa647b6378414543a562e9e3c83dcbf44bcb232d021590"} Nov 03 17:41:35.560093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:35.557610 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podStartSLOduration=2.557587913 podStartE2EDuration="2.557587913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:35.530848298 +0000 UTC m=+1437.985537324" watchObservedRunningTime="2026-11-03 17:41:35.557587913 +0000 UTC m=+1438.012276941" Nov 03 17:41:36.127662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.127585 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:36.129816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.129786 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:36.130310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.130284 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:36.179311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.179273 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:36.187528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.186594 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:36.187528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:36.186651 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:37.127561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.127522 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:37.127828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.127795 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:37.175466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.175437 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:37.175691 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.175670 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:37.973395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.973361 4885 scope.go:122] "RemoveContainer" containerID="44b31a549908c17e4f038185955f4be0e91a03038c67bda24845921640739f13" Nov 03 17:41:37.982475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:37.982454 4885 scope.go:122] "RemoveContainer" containerID="371df12697f3b548926b977acf29c6919068a8f7d25e2aa56445cd49b411230e" Nov 03 17:41:38.141090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:38.140450 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:38.141090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:38.140522 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:38.175876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:38.175843 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:38.176049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:38.175892 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:39.125516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:39.125474 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:39.125824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:39.125795 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:39.176004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:39.175972 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:39.176408 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:39.176037 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:40.128094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:40.125443 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:40.128094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:40.125523 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:40.175999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:40.175959 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:40.176203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:40.176033 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:40.359108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:40.359042 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:41:41.129095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:41.128490 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:41.129095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:41.128561 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:41.175521 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:41.175490 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:41.175683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:41.175538 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:42.125578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:42.125541 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:42.126102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:42.125607 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:42.175872 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:42.175834 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:42.176100 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:42.175913 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:43.125409 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:43.125366 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:41:43.125616 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:43.125423 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:41:43.175384 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:43.175344 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:43.175571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:43.175413 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:44.176633 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:44.176598 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-c6nsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 17:41:44.177253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:44.177220 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 17:41:45.197290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.197240 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:41:45.197837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.197311 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:41:45.374481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.374445 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:45.375749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.375727 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:45.376579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.376560 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:41:45.500679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.500547 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podStartSLOduration=12.500529703 podStartE2EDuration="12.500529703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:35.658031798 +0000 UTC m=+1438.112720824" watchObservedRunningTime="2026-11-03 17:41:45.500529703 +0000 UTC m=+1447.955218730" Nov 03 17:41:45.671315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.671283 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:45.673159 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.673136 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:45.674480 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:45.674435 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:41:48.605906 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:41:48.605871 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:41:50.342469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:50.342432 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:41:50.347447 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:50.347424 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:41:53.072992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:53.072963 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-67bfdd8c49-4rgpl_06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df/payload-processing/0.log" Nov 03 17:41:56.622124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.621390 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8"] Nov 03 17:41:56.638385 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.636021 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.666431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.666400 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8"] Nov 03 17:41:56.678609 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.678576 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4gl\" (UniqueName: \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.678789 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.678641 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.678789 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.678708 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.690499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.690469 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:41:56.713485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.713461 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:41:56.713485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.713472 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:41:56.713694 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.713544 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:41:56.780149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.780114 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4gl\" (UniqueName: \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.780308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.780176 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.780308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.780204 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.781037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.781015 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.782945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.782923 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.844422 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.844396 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4gl\" (UniqueName: \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\") pod \"maas-api-revoke-sub-685e356a-2ldp8\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:56.956515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:56.956435 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:41:57.336695 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:41:57.336664 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81355d5a_7d30_411b_bf81_8a600742400e.slice/crio-2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5 WatchSource:0}: Error finding container 2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5: Status 404 returned error can't find the container with id 2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5 Nov 03 17:41:57.337328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:57.337305 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8"] Nov 03 17:41:57.710214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:57.710179 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" event={"ID":"81355d5a-7d30-411b-bf81-8a600742400e","Type":"ContainerStarted","Data":"33612f25920eb098a871f0fcb421c5aa3ceb5d5647b9adbac9ce770455857187"} Nov 03 17:41:57.710638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:57.710235 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" event={"ID":"81355d5a-7d30-411b-bf81-8a600742400e","Type":"ContainerStarted","Data":"2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5"} Nov 03 17:41:57.831539 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:57.831490 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" podStartSLOduration=1.831476643 podStartE2EDuration="1.831476643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:57.829449187 +0000 UTC m=+1460.284138211" watchObservedRunningTime="2026-11-03 17:41:57.831476643 +0000 UTC m=+1460.286165665" Nov 03 17:41:58.412258 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.412176 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:41:58.416598 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.416575 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.481513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.481477 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:41:58.498968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.498941 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.499135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.498999 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.499200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.499128 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8bw\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.499307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.499287 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.519506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.519478 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\"" Nov 03 17:41:58.519506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.519499 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-18e023f84819\"" Nov 03 17:41:58.519707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.519491 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\"" Nov 03 17:41:58.519707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.519525 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\"" Nov 03 17:41:58.519707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.519604 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\"" Nov 03 17:41:58.522252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.522232 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\"" Nov 03 17:41:58.522252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.522248 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\"" Nov 03 17:41:58.522633 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.522617 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\"" Nov 03 17:41:58.522684 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.522631 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\"" Nov 03 17:41:58.600902 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.600869 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.601153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.600945 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.601153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.601008 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8bw\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.601153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.601139 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.601749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.601729 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.602006 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.601952 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.604020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.604002 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.663852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.663776 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8bw\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:58.717509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.717478 4885 generic.go:356] "Generic (PLEG): container finished" podID="81355d5a-7d30-411b-bf81-8a600742400e" containerID="33612f25920eb098a871f0fcb421c5aa3ceb5d5647b9adbac9ce770455857187" exitCode=0 Nov 03 17:41:58.717882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.717520 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" event={"ID":"81355d5a-7d30-411b-bf81-8a600742400e","Type":"ContainerDied","Data":"33612f25920eb098a871f0fcb421c5aa3ceb5d5647b9adbac9ce770455857187"} Nov 03 17:41:58.733351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:58.733322 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:41:59.121801 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:41:59.121770 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82b6fdc_9bfe_4968_9cbb_107b8c94c3ba.slice/crio-7e0cc3569d8fb1cbbea933d69a79b3787021ebd286eff42a215385c891394dfa WatchSource:0}: Error finding container 7e0cc3569d8fb1cbbea933d69a79b3787021ebd286eff42a215385c891394dfa: Status 404 returned error can't find the container with id 7e0cc3569d8fb1cbbea933d69a79b3787021ebd286eff42a215385c891394dfa Nov 03 17:41:59.124670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:59.124553 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:41:59.725264 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:59.725228 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" event={"ID":"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba","Type":"ContainerStarted","Data":"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93"} Nov 03 17:41:59.725768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:59.725285 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" event={"ID":"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba","Type":"ContainerStarted","Data":"7e0cc3569d8fb1cbbea933d69a79b3787021ebd286eff42a215385c891394dfa"} Nov 03 17:41:59.849522 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:59.849472 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" podStartSLOduration=1.849458159 podStartE2EDuration="1.849458159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:59.84674685 +0000 UTC m=+1462.301435876" watchObservedRunningTime="2026-11-03 17:41:59.849458159 +0000 UTC m=+1462.304147200" Nov 03 17:41:59.905676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:41:59.905649 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:42:00.019200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.019105 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\") pod \"81355d5a-7d30-411b-bf81-8a600742400e\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " Nov 03 17:42:00.019386 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.019228 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\" (UniqueName: \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\") pod \"81355d5a-7d30-411b-bf81-8a600742400e\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " Nov 03 17:42:00.019386 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.019286 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\") pod \"81355d5a-7d30-411b-bf81-8a600742400e\" (UID: \"81355d5a-7d30-411b-bf81-8a600742400e\") " Nov 03 17:42:00.019727 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.019697 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca" pod "81355d5a-7d30-411b-bf81-8a600742400e" (UID: "81355d5a-7d30-411b-bf81-8a600742400e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:00.019939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.019854 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:00.022607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.022575 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca" pod "81355d5a-7d30-411b-bf81-8a600742400e" (UID: "81355d5a-7d30-411b-bf81-8a600742400e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:00.023466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.023404 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl" pod "81355d5a-7d30-411b-bf81-8a600742400e" (UID: "81355d5a-7d30-411b-bf81-8a600742400e"). InnerVolumeSpecName "kube-api-access-8b4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:00.121400 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.121364 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81355d5a-7d30-411b-bf81-8a600742400e-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:00.121400 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.121398 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4gl\" (UniqueName: \"kubernetes.io/projected/81355d5a-7d30-411b-bf81-8a600742400e-kube-api-access-8b4gl\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:00.342598 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.342518 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:00.732746 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.732709 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" event={"ID":"81355d5a-7d30-411b-bf81-8a600742400e","Type":"ContainerDied","Data":"2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5"} Nov 03 17:42:00.733244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.732773 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d018f227068e4a45451345c30a635aefaf28288e0818c3d893ffc8969ff9fc5" Nov 03 17:42:00.733244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.732729 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8" Nov 03 17:42:00.733244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:00.732904 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:42:06.053709 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:06.053670 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4"] Nov 03 17:42:06.083513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:06.083484 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960c2fb0-l62z4"] Nov 03 17:42:07.324169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.324131 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb"] Nov 03 17:42:07.324856 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.324839 4885 cpu_manager.go:414] "containerMap: removing container" podUID="81355d5a-7d30-411b-bf81-8a600742400e" containerName="revoke-keys" Nov 03 17:42:07.324914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.324860 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81355d5a-7d30-411b-bf81-8a600742400e" containerName="revoke-keys" Nov 03 17:42:07.325009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.324989 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="81355d5a-7d30-411b-bf81-8a600742400e" containerName="revoke-keys" Nov 03 17:42:07.328350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.328334 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.380519 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.380492 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb"] Nov 03 17:42:07.421989 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.421959 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:42:07.429183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.429162 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:42:07.429280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.429240 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:42:07.429280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.429252 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:42:07.452296 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.452275 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.452407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.452318 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5f7\" (UniqueName: \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.452407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.452352 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.553820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.553789 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.554002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.553920 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.554002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.553954 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5f7\" (UniqueName: \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.554570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.554551 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.556348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.556327 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.616756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.616687 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5f7\" (UniqueName: \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\") pod \"maas-api-revoke-sub-cfac8996-k4tcb\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.642160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.642132 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:07.665306 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:07.665275 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7324d4-fff3-4ee8-9117-d318afdfe2c7" path="/var/lib/kubelet/pods/ee7324d4-fff3-4ee8-9117-d318afdfe2c7/volumes" Nov 03 17:42:08.029981 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.029955 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb"] Nov 03 17:42:08.600662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.600633 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:42:08.603254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.603231 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:42:08.862461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.862371 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" event={"ID":"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f","Type":"ContainerStarted","Data":"86be1b0a1c01c207007b996916d3c73a921b0889b2274215472b84f403f59da7"} Nov 03 17:42:08.862461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.862433 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" event={"ID":"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f","Type":"ContainerStarted","Data":"8a7bb5eb83cda555fcf1d27f09cc3889172c722b8635f2f1a2a66e7abca0ba89"} Nov 03 17:42:08.983945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:08.983901 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" podStartSLOduration=1.9838875489999999 podStartE2EDuration="1.983887549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:08.982248144 +0000 UTC m=+1471.436937168" watchObservedRunningTime="2026-11-03 17:42:08.983887549 +0000 UTC m=+1471.438576574" Nov 03 17:42:09.869573 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:09.869538 4885 generic.go:356] "Generic (PLEG): container finished" podID="cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" containerID="86be1b0a1c01c207007b996916d3c73a921b0889b2274215472b84f403f59da7" exitCode=0 Nov 03 17:42:09.869978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:09.869605 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" event={"ID":"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f","Type":"ContainerDied","Data":"86be1b0a1c01c207007b996916d3c73a921b0889b2274215472b84f403f59da7"} Nov 03 17:42:10.342240 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:10.342158 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:10.974492 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:10.974460 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:42:10.975880 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:10.975838 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:11.046337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.046312 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:11.116404 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.116366 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\" (UniqueName: \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\") pod \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " Nov 03 17:42:11.116605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.116465 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\") pod \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " Nov 03 17:42:11.116605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.116525 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\") pod \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\" (UID: \"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f\") " Nov 03 17:42:11.117055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.117031 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca" pod "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" (UID: "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:11.119001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.118977 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca" pod "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" (UID: "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:11.119087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.119022 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7" pod "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" (UID: "cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f"). InnerVolumeSpecName "kube-api-access-6h5f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:11.218331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.218295 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h5f7\" (UniqueName: \"kubernetes.io/projected/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-kube-api-access-6h5f7\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:11.218331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.218330 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:11.218568 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.218345 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:11.882561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.882536 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" Nov 03 17:42:11.882561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.882552 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb" event={"ID":"cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f","Type":"ContainerDied","Data":"8a7bb5eb83cda555fcf1d27f09cc3889172c722b8635f2f1a2a66e7abca0ba89"} Nov 03 17:42:11.882756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:11.882603 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a7bb5eb83cda555fcf1d27f09cc3889172c722b8635f2f1a2a66e7abca0ba89" Nov 03 17:42:13.602013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.601951 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" podUID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" containerName="payload-processing" containerID="cri-o://ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93" gracePeriod=25 Nov 03 17:42:13.870934 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.870910 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:42:13.894699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.894667 4885 generic.go:356] "Generic (PLEG): container finished" podID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" containerID="ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93" exitCode=0 Nov 03 17:42:13.894827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.894721 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" Nov 03 17:42:13.894827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.894753 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" event={"ID":"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba","Type":"ContainerDied","Data":"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93"} Nov 03 17:42:13.894827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.894804 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr" event={"ID":"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba","Type":"ContainerDied","Data":"7e0cc3569d8fb1cbbea933d69a79b3787021ebd286eff42a215385c891394dfa"} Nov 03 17:42:13.894968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.894829 4885 scope.go:122] "RemoveContainer" containerID="ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93" Nov 03 17:42:13.906337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.906192 4885 scope.go:122] "RemoveContainer" containerID="ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93" Nov 03 17:42:13.906603 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:42:13.906566 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93\": container with ID starting with ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93 not found: ID does not exist" containerID="ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93" Nov 03 17:42:13.906716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.906608 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93"} err="failed to get container status \"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93\": rpc error: code = NotFound desc = could not find container \"ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93\": container with ID starting with ee3df54fd7545d5074e53e5d0fefa861b6404106779bb3d1d3542c4a9d6d3a93 not found: ID does not exist" Nov 03 17:42:13.949549 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.949522 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\") pod \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " Nov 03 17:42:13.949684 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.949649 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\") pod \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " Nov 03 17:42:13.949743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.949728 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\") pod \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " Nov 03 17:42:13.949854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.949839 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\") pod \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\" (UID: \"c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba\") " Nov 03 17:42:13.950155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.950126 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume" pod "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" (UID: "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:13.950255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.950220 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume" pod "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" (UID: "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:13.950446 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.950291 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:13.951939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.951912 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials" pod "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" (UID: "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:13.951939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:13.951927 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw" pod "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" (UID: "c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba"). InnerVolumeSpecName "kube-api-access-zc8bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:14.051081 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:14.051037 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8bw\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-kube-api-access-zc8bw\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:14.051236 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:14.051091 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:14.051236 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:14.051102 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:14.320329 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:14.320296 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:42:14.325205 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:14.325181 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-74bbn6gdr"] Nov 03 17:42:15.661608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:15.661574 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" path="/var/lib/kubelet/pods/c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba/volumes" Nov 03 17:42:15.975634 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:15.975533 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" containerID="cri-o://dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52" gracePeriod=25 Nov 03 17:42:16.228964 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.228901 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:42:16.379234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379202 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\") pod \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " Nov 03 17:42:16.379401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379278 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\") pod \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " Nov 03 17:42:16.379401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379367 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\") pod \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " Nov 03 17:42:16.379516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379429 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\") pod \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\" (UID: \"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df\") " Nov 03 17:42:16.381815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379796 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume" pod "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" (UID: "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:16.381815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.379887 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:16.381815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.380115 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume" pod "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" (UID: "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:16.382343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.382317 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht" pod "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" (UID: "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df"). InnerVolumeSpecName "kube-api-access-k8sht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:16.382512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.382492 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials" pod "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" (UID: "06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:16.481738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.481646 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:16.481738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.481680 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sht\" (UniqueName: \"kubernetes.io/projected/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-kube-api-access-k8sht\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:16.481738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.481694 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:16.912502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.912464 4885 generic.go:356] "Generic (PLEG): container finished" podID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerID="dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52" exitCode=0 Nov 03 17:42:16.912945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.912534 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" Nov 03 17:42:16.912945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.912542 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" event={"ID":"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df","Type":"ContainerDied","Data":"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52"} Nov 03 17:42:16.912945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.912605 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl" event={"ID":"06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df","Type":"ContainerDied","Data":"0ab96eecf6446be21c20ae5dcee3b10aade31d2fdad90a736dbe10471a0919e6"} Nov 03 17:42:16.912945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.912629 4885 scope.go:122] "RemoveContainer" containerID="dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52" Nov 03 17:42:16.932421 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.932401 4885 scope.go:122] "RemoveContainer" containerID="dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52" Nov 03 17:42:16.932734 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:42:16.932709 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52\": container with ID starting with dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52 not found: ID does not exist" containerID="dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52" Nov 03 17:42:16.932781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:16.932750 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52"} err="failed to get container status \"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52\": rpc error: code = NotFound desc = could not find container \"dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52\": container with ID starting with dafb2d78d8bb1fcf26dc4f64f57d151133e52ba078867c75a7e51c6b258e8e52 not found: ID does not exist" Nov 03 17:42:17.083451 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:17.083416 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:42:17.134530 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:17.134501 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67bfdd8c49-4rgpl"] Nov 03 17:42:17.663234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:17.663204 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" path="/var/lib/kubelet/pods/06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df/volumes" Nov 03 17:42:18.606029 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:42:18.605979 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:42:20.492652 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.492622 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b"] Nov 03 17:42:20.493054 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493042 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" containerName="payload-processing" Nov 03 17:42:20.493130 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493055 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" containerName="payload-processing" Nov 03 17:42:20.493167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493136 4885 cpu_manager.go:414] "containerMap: removing container" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" Nov 03 17:42:20.493167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493144 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" Nov 03 17:42:20.493167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493152 4885 cpu_manager.go:414] "containerMap: removing container" podUID="cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" containerName="revoke-keys" Nov 03 17:42:20.493167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493157 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" containerName="revoke-keys" Nov 03 17:42:20.493282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493234 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ff6d4d-cf62-4f0f-bc38-dc6c6988f7df" containerName="payload-processing" Nov 03 17:42:20.493282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493241 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" containerName="revoke-keys" Nov 03 17:42:20.493282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.493248 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82b6fdc-9bfe-4968-9cbb-107b8c94c3ba" containerName="payload-processing" Nov 03 17:42:20.497971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.497955 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.548821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.548793 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz58d\" (UniqueName: \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.549014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.548829 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.549014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.548994 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.550545 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.550515 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b"] Nov 03 17:42:20.598508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.598478 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:42:20.598508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.598478 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:42:20.598698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.598554 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:42:20.604311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.604289 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:42:20.649878 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.649852 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.650039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.649954 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz58d\" (UniqueName: \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.650039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.649981 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.650606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.650577 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.652571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.652551 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.712873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.712843 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz58d\" (UniqueName: \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\") pod \"maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:20.812512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:20.812416 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:21.182860 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:42:21.182822 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174af691_d0e2_41b6_849b_40760c032323.slice/crio-e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22 WatchSource:0}: Error finding container e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22: Status 404 returned error can't find the container with id e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22 Nov 03 17:42:21.185173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:21.185147 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b"] Nov 03 17:42:21.948048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:21.948010 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" event={"ID":"174af691-d0e2-41b6-849b-40760c032323","Type":"ContainerStarted","Data":"a1f009f9bb711ba1a1ad73dff4b4bd4066b972fee23482195cce8e4fe8e3c49b"} Nov 03 17:42:21.948427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:21.948115 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" event={"ID":"174af691-d0e2-41b6-849b-40760c032323","Type":"ContainerStarted","Data":"e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22"} Nov 03 17:42:22.055693 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:22.055664 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm"] Nov 03 17:42:22.057940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:22.057916 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba755f0-stbfm"] Nov 03 17:42:22.099227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:22.099176 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" podStartSLOduration=2.099165357 podStartE2EDuration="2.099165357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:22.098930902 +0000 UTC m=+1484.553619926" watchObservedRunningTime="2026-11-03 17:42:22.099165357 +0000 UTC m=+1484.553854380" Nov 03 17:42:22.961319 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:22.961284 4885 generic.go:356] "Generic (PLEG): container finished" podID="174af691-d0e2-41b6-849b-40760c032323" containerID="a1f009f9bb711ba1a1ad73dff4b4bd4066b972fee23482195cce8e4fe8e3c49b" exitCode=0 Nov 03 17:42:22.965180 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:22.962494 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" event={"ID":"174af691-d0e2-41b6-849b-40760c032323","Type":"ContainerDied","Data":"a1f009f9bb711ba1a1ad73dff4b4bd4066b972fee23482195cce8e4fe8e3c49b"} Nov 03 17:42:23.663021 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:23.662987 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46205b2f-f17a-49ad-bfae-bbdb740c4a59" path="/var/lib/kubelet/pods/46205b2f-f17a-49ad-bfae-bbdb740c4a59/volumes" Nov 03 17:42:24.117037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.117010 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:24.208698 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.208666 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\") pod \"174af691-d0e2-41b6-849b-40760c032323\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " Nov 03 17:42:24.208860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.208712 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\") pod \"174af691-d0e2-41b6-849b-40760c032323\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " Nov 03 17:42:24.208860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.208808 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\" (UniqueName: \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\") pod \"174af691-d0e2-41b6-849b-40760c032323\" (UID: \"174af691-d0e2-41b6-849b-40760c032323\") " Nov 03 17:42:24.209169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.209122 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca" pod "174af691-d0e2-41b6-849b-40760c032323" (UID: "174af691-d0e2-41b6-849b-40760c032323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:24.209348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.209329 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/174af691-d0e2-41b6-849b-40760c032323-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.211364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.211342 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca" pod "174af691-d0e2-41b6-849b-40760c032323" (UID: "174af691-d0e2-41b6-849b-40760c032323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:24.213223 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.213202 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d" pod "174af691-d0e2-41b6-849b-40760c032323" (UID: "174af691-d0e2-41b6-849b-40760c032323"). InnerVolumeSpecName "kube-api-access-vz58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:24.310024 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.309945 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/174af691-d0e2-41b6-849b-40760c032323-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.310024 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.309980 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz58d\" (UniqueName: \"kubernetes.io/projected/174af691-d0e2-41b6-849b-40760c032323-kube-api-access-vz58d\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.978540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.978510 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" Nov 03 17:42:24.978540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.978518 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b" event={"ID":"174af691-d0e2-41b6-849b-40760c032323","Type":"ContainerDied","Data":"e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22"} Nov 03 17:42:24.978747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:24.978561 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e52f7ee5366ef2a6ad1b58584d9098e27b9fd98924a7c571f529011599b03f22" Nov 03 17:42:27.662640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.662605 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:42:27.663035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.663013 4885 cpu_manager.go:414] "containerMap: removing container" podUID="174af691-d0e2-41b6-849b-40760c032323" containerName="revoke-keys" Nov 03 17:42:27.663035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.663025 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174af691-d0e2-41b6-849b-40760c032323" containerName="revoke-keys" Nov 03 17:42:27.663159 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.663145 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="174af691-d0e2-41b6-849b-40760c032323" containerName="revoke-keys" Nov 03 17:42:27.666370 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.666354 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.722402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.722370 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:42:27.747806 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.747761 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.747999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.747898 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgggn\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.747999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.747951 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.748173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.748142 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.769111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.769017 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\"" Nov 03 17:42:27.769111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.769036 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\"" Nov 03 17:42:27.769111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.769041 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\"" Nov 03 17:42:27.769111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.769108 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\"" Nov 03 17:42:27.769365 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.769165 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\"" Nov 03 17:42:27.774141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.774121 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\"" Nov 03 17:42:27.774141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.774123 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-d27edff159f1\"" Nov 03 17:42:27.774269 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.774160 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\"" Nov 03 17:42:27.774269 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.774172 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\"" Nov 03 17:42:27.849687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.849651 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.849968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.849707 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgggn\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.849968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.849738 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.849968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.849827 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.850707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.850670 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.850871 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.850704 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.852924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.852901 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.912678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.912652 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgggn\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:27.979509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:27.979475 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:28.364832 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:42:28.364801 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8c273e_6a53_418d_b0ec_a94c4c71eb3c.slice/crio-514e0645ae1126cfc115aab4095551059267aef37b966bb32329eea8081bb142 WatchSource:0}: Error finding container 514e0645ae1126cfc115aab4095551059267aef37b966bb32329eea8081bb142: Status 404 returned error can't find the container with id 514e0645ae1126cfc115aab4095551059267aef37b966bb32329eea8081bb142 Nov 03 17:42:28.368212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:28.368187 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:42:28.852378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:28.852336 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:42:29.003279 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:29.003245 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" event={"ID":"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c","Type":"ContainerStarted","Data":"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70"} Nov 03 17:42:29.003450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:29.003301 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" event={"ID":"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c","Type":"ContainerStarted","Data":"514e0645ae1126cfc115aab4095551059267aef37b966bb32329eea8081bb142"} Nov 03 17:42:29.003450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:29.003353 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:29.142222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:29.142161 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" podStartSLOduration=2.142144084 podStartE2EDuration="2.142144084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:29.139660073 +0000 UTC m=+1491.594349090" watchObservedRunningTime="2026-11-03 17:42:29.142144084 +0000 UTC m=+1491.596833112" Nov 03 17:42:40.029711 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:40.029672 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:42:48.605737 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:42:48.605698 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:42:54.416185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:54.416139 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:42:54.463203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:54.463159 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:42:54.634254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:54.634207 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:42:54.634575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:54.634525 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:42:59.919870 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:59.919833 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b"] Nov 03 17:42:59.975477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:42:59.975443 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f88f673-fh78b"] Nov 03 17:43:01.702605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:01.702566 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174af691-d0e2-41b6-849b-40760c032323" path="/var/lib/kubelet/pods/174af691-d0e2-41b6-849b-40760c032323/volumes" Nov 03 17:43:05.823253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:05.823201 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:43:11.330484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:11.330440 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:43:12.082274 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:12.081138 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 03 17:43:18.605586 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:43:18.605554 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:43:19.063360 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:19.063328 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5"] Nov 03 17:43:19.113896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:19.113866 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a9d869-w2vg5"] Nov 03 17:43:19.663594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:19.663556 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59222f9e-f097-4ad3-bafb-54d2afd7a7fa" path="/var/lib/kubelet/pods/59222f9e-f097-4ad3-bafb-54d2afd7a7fa/volumes" Nov 03 17:43:26.826999 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.826965 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:43:26.830973 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.830955 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.932805 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.932765 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.932990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.932824 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.932990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.932935 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933012 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbhp\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933055 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933145 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933185 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933231 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933211 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.933371 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.933343 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:26.938790 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:26.938764 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:43:27.034433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034395 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbhp\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034435 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034478 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034507 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034530 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034638 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034672 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034705 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.034940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.034744 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.035047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.035008 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.035234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.035183 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.035367 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.035257 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.035367 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.035296 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.035367 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.035334 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.037650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.037629 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.037860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.037837 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.097195 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.097116 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.097195 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.097182 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbhp\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\") pod \"e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.145846 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.145812 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:27.569092 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:43:27.569029 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768929a4_275d_49b7_9356_17aedbbfb25c.slice/crio-f2d5d13eb023208636114ce16a3c30f9b7298529c6efae9e99285c6c533dcc13 WatchSource:0}: Error finding container f2d5d13eb023208636114ce16a3c30f9b7298529c6efae9e99285c6c533dcc13: Status 404 returned error can't find the container with id f2d5d13eb023208636114ce16a3c30f9b7298529c6efae9e99285c6c533dcc13 Nov 03 17:43:27.571438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.571404 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:43:27.571542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.571467 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:43:27.571542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.571496 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:43:27.572014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.571995 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:43:27.842143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.838250 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" event={"ID":"768929a4-275d-49b7-9356-17aedbbfb25c","Type":"ContainerStarted","Data":"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf"} Nov 03 17:43:27.842143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.838355 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" event={"ID":"768929a4-275d-49b7-9356-17aedbbfb25c","Type":"ContainerStarted","Data":"f2d5d13eb023208636114ce16a3c30f9b7298529c6efae9e99285c6c533dcc13"} Nov 03 17:43:27.964420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:27.964359 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podStartSLOduration=1.964342257 podStartE2EDuration="1.964342257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:27.963190078 +0000 UTC m=+1550.417879101" watchObservedRunningTime="2026-11-03 17:43:27.964342257 +0000 UTC m=+1550.419031280" Nov 03 17:43:28.146344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:28.146306 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:28.147441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:28.147416 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:28.147579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:28.147479 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:29.147361 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:29.147270 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:29.147361 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:29.147346 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:30.146546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:30.146510 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:30.146834 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:30.146797 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:31.147127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:31.147099 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:31.147487 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:31.147158 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:32.070560 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:32.070523 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w"] Nov 03 17:43:32.128091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:32.127772 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5734ce0c-d6r8w"] Nov 03 17:43:32.147120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:32.147083 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:32.147307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:32.147140 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:33.147172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:33.147138 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:43:33.147350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:33.147201 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 17:43:33.662803 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:33.662769 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75172a92-cf86-4a50-97d0-f3ad832b7c46" path="/var/lib/kubelet/pods/75172a92-cf86-4a50-97d0-f3ad832b7c46/volumes" Nov 03 17:43:34.271420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:34.271390 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:34.271879 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:34.271706 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:34.272684 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:34.272666 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:43:37.271253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.271213 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:43:37.276047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.276018 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.380666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.380634 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"openshift-service-ca.crt\"" Nov 03 17:43:37.382280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382410 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382315 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382451 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382415 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382464 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382551 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382537 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382593 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5d7h\" (UniqueName: \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.382888 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382873 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"kube-root-ca.crt\"" Nov 03 17:43:37.382990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.382932 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"sub-test-model-147834-kserve-self-signed-certs\"" Nov 03 17:43:37.383508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.383494 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-147834\"/\"default-dockercfg-78pkk\"" Nov 03 17:43:37.390268 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.390252 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:43:37.485835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.485791 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.486174 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.486148 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.486406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.486362 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.486569 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.486553 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.486752 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.486734 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.486894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.486876 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5d7h\" (UniqueName: \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.487831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.487808 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.488889 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.488870 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.489338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.489288 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.510783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.510754 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.524798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.524720 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.552565 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.552529 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5d7h\" (UniqueName: \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\") pod \"sub-test-model-147834-kserve-cc6c97b7f-5rnjk\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:37.588774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:37.588747 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:38.050699 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:43:38.050675 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfac6a0b8_28ed_458f_ba98_c76124ab5499.slice/crio-3db65ab582b0d9d927e9473582743d9b632a3569d71b603a990647e48c34ec68 WatchSource:0}: Error finding container 3db65ab582b0d9d927e9473582743d9b632a3569d71b603a990647e48c34ec68: Status 404 returned error can't find the container with id 3db65ab582b0d9d927e9473582743d9b632a3569d71b603a990647e48c34ec68 Nov 03 17:43:38.051865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:38.051839 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:43:38.071357 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:38.071337 4885 scope.go:122] "RemoveContainer" containerID="48d6531aeb35f84bc917c9aee468ad36cc989f3dc52e06b4d51b2c045f2fab71" Nov 03 17:43:38.083285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:38.083264 4885 scope.go:122] "RemoveContainer" containerID="bb6ab4fb6db65a001de2d0f5346a006d8cb3bb633a44522bcf702258855558b9" Nov 03 17:43:38.919753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:38.919715 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" event={"ID":"fac6a0b8-28ed-458f-ba98-c76124ab5499","Type":"ContainerStarted","Data":"bc03386c1141cbe74a6cb7cfe705aef55da6187f4258eb10113098d78751dd0d"} Nov 03 17:43:38.920187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:38.919780 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" event={"ID":"fac6a0b8-28ed-458f-ba98-c76124ab5499","Type":"ContainerStarted","Data":"3db65ab582b0d9d927e9473582743d9b632a3569d71b603a990647e48c34ec68"} Nov 03 17:43:39.051812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:39.051758 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podStartSLOduration=2.051744701 podStartE2EDuration="2.051744701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:39.049316236 +0000 UTC m=+1561.504005259" watchObservedRunningTime="2026-11-03 17:43:39.051744701 +0000 UTC m=+1561.506433724" Nov 03 17:43:39.589184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:39.589151 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:43.159727 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.159680 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:43:43.163271 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.163250 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.258583 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.258548 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.258759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.258714 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.258824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.258775 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.258864 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.258834 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7xt\" (UniqueName: \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.271042 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.271017 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0b55a36f-serving-cert\"" Nov 03 17:43:43.271227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.271048 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:43:43.271227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.271059 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:43:43.271227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.271104 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:43:43.271227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.271151 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:43:43.273686 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.273662 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:43:43.360110 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.360055 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.360273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.360170 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.360273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.360203 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7xt\" (UniqueName: \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.360348 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.360268 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.360348 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:43:43.360325 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-serving-cert: secret "maas-api-e2e-ait-0b55a36f-serving-cert" not found Nov 03 17:43:43.360424 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:43:43.360404 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls podName:a3b4e81a-060a-4e6d-8b74-183f72a19b17 nodeName:}" failed. No retries permitted until 2026-11-03 17:43:43.860384225 +0000 UTC m=+1566.315073247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls") pod "maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" (UID: "a3b4e81a-060a-4e6d-8b74-183f72a19b17") : secret "maas-api-e2e-ait-0b55a36f-serving-cert" not found Nov 03 17:43:43.360959 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.360940 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.362556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.362539 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.474141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.468750 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7xt\" (UniqueName: \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.873300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.873268 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:43.876015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:43.875990 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") pod \"maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:44.076215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:44.076188 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:44.258814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:44.257170 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:43:44.507990 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:43:44.507957 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b4e81a_060a_4e6d_8b74_183f72a19b17.slice/crio-29d901b7c61d29477b43f6e5e1d893cc0922a761557c6be1498d65e0e0e95f1f WatchSource:0}: Error finding container 29d901b7c61d29477b43f6e5e1d893cc0922a761557c6be1498d65e0e0e95f1f: Status 404 returned error can't find the container with id 29d901b7c61d29477b43f6e5e1d893cc0922a761557c6be1498d65e0e0e95f1f Nov 03 17:43:44.510743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:44.510717 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:43:44.963090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:44.962783 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" event={"ID":"a3b4e81a-060a-4e6d-8b74-183f72a19b17","Type":"ContainerStarted","Data":"29d901b7c61d29477b43f6e5e1d893cc0922a761557c6be1498d65e0e0e95f1f"} Nov 03 17:43:45.969808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:45.969768 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" event={"ID":"a3b4e81a-060a-4e6d-8b74-183f72a19b17","Type":"ContainerStarted","Data":"22d314df936b0caaca200defab0ca12d9c46a2f3b3b85c44cdc30a2591fb8959"} Nov 03 17:43:45.970275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:45.969827 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:43:46.092651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:46.092600 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" podStartSLOduration=2.068080259 podStartE2EDuration="3.092584738s" totalImagesPullingTime="1.024504479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:46.091033175 +0000 UTC m=+1568.545722198" watchObservedRunningTime="2026-11-03 17:43:46.092584738 +0000 UTC m=+1568.547273762" Nov 03 17:43:47.589824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:47.589793 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:47.603043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:47.603009 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:47.992185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:47.992154 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:43:48.605984 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:43:48.605940 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:43:49.056494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:49.056459 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9"] Nov 03 17:43:49.109113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:49.109055 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a401a1c-l9lt9"] Nov 03 17:43:49.661924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:49.661892 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0d1cfe-c14f-4779-9375-a8636cae4e02" path="/var/lib/kubelet/pods/bb0d1cfe-c14f-4779-9375-a8636cae4e02/volumes" Nov 03 17:43:51.981438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:43:51.981411 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:44:14.679207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:14.679167 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:44:18.604959 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:44:18.604925 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:44:31.058294 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:31.058257 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js"] Nov 03 17:44:31.110805 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:31.110772 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f41ec4e-2x7js"] Nov 03 17:44:31.662007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:31.661963 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad0c6f4-4f4f-40bc-ad13-09f22aac4392" path="/var/lib/kubelet/pods/2ad0c6f4-4f4f-40bc-ad13-09f22aac4392/volumes" Nov 03 17:44:38.112651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:38.112622 4885 scope.go:122] "RemoveContainer" containerID="e3622186581088832e104e11be5d8932dbe8103bb055c45cf73243d4dbc78cf1" Nov 03 17:44:38.122061 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:38.122039 4885 scope.go:122] "RemoveContainer" containerID="be327d78f01c2f850848330414b6774e875fbcb4676065c3f1af626aca64eed3" Nov 03 17:44:48.607122 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:44:48.607056 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:44:50.728043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:44:50.727123 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:45:00.063848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.063776 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:45:00.064298 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.063959 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" podUID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" containerName="maas-api" containerID="cri-o://22d314df936b0caaca200defab0ca12d9c46a2f3b3b85c44cdc30a2591fb8959" gracePeriod=30 Nov 03 17:45:00.220232 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.220193 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:00.224807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.224778 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.227331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.227302 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:00.232163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.232137 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.444407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.444376 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpnx4\" (UniqueName: \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.444734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.444710 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.445036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.444364 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:00.453525 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.444993 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zsv\" (UniqueName: \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.453622 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.453581 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.464474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.464450 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.465328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.465294 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:45:00.467409 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.467389 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p"] Nov 03 17:45:00.471447 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.471429 4885 generic.go:356] "Generic (PLEG): container finished" podID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" containerID="22d314df936b0caaca200defab0ca12d9c46a2f3b3b85c44cdc30a2591fb8959" exitCode=0 Nov 03 17:45:00.472049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.472038 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:00.472193 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.472180 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" event={"ID":"a3b4e81a-060a-4e6d-8b74-183f72a19b17","Type":"ContainerDied","Data":"22d314df936b0caaca200defab0ca12d9c46a2f3b3b85c44cdc30a2591fb8959"} Nov 03 17:45:00.472314 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.472304 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.476176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.476106 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:00.506170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.506136 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p"] Nov 03 17:45:00.597092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.575313 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.603304 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.603273 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpcwq\" (UniqueName: \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.640116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.613233 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.647464 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.644266 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpnx4\" (UniqueName: \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.647733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.647706 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.647873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.647859 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zsv\" (UniqueName: \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.648009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.647992 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmz2d\" (UniqueName: \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.648180 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.648137 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.651726 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:00.651677 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jpcwq maas-api-service-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" podUID="48bfefa6-8a12-4c3c-af1e-e382c45afe5e" Nov 03 17:45:00.651726 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.602116 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p"] Nov 03 17:45:00.651963 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.651782 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:00.654890 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.654849 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.655645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.655528 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.709979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.709638 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpnx4\" (UniqueName: \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\") pod \"maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.710873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.710851 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:45:00.716213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.716192 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zsv\" (UniqueName: \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:00.733710 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.733684 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:00.749650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.749624 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.749804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.749671 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcwq\" (UniqueName: \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.749804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.749737 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.749922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.749869 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmz2d\" (UniqueName: \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.750811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.750782 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.750922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.750825 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.817568 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.817503 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmz2d\" (UniqueName: \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.819111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.818259 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcwq\" (UniqueName: \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\") pod \"maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:00.850787 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.850742 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\") pod \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " Nov 03 17:45:00.850949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.850898 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\") pod \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " Nov 03 17:45:00.851033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.851019 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\" (UniqueName: \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\") pod \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " Nov 03 17:45:00.851161 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.851093 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") pod \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\" (UID: \"a3b4e81a-060a-4e6d-8b74-183f72a19b17\") " Nov 03 17:45:00.852146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.852108 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca" pod "a3b4e81a-060a-4e6d-8b74-183f72a19b17" (UID: "a3b4e81a-060a-4e6d-8b74-183f72a19b17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:00.854089 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.854032 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls" pod "a3b4e81a-060a-4e6d-8b74-183f72a19b17" (UID: "a3b4e81a-060a-4e6d-8b74-183f72a19b17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:00.854504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.854477 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt" pod "a3b4e81a-060a-4e6d-8b74-183f72a19b17" (UID: "a3b4e81a-060a-4e6d-8b74-183f72a19b17"). InnerVolumeSpecName "kube-api-access-4n7xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:00.855742 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.855709 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs" pod "a3b4e81a-060a-4e6d-8b74-183f72a19b17" (UID: "a3b4e81a-060a-4e6d-8b74-183f72a19b17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:00.880533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.880506 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:00.953474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.953382 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b4e81a-060a-4e6d-8b74-183f72a19b17-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:00.953474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.953420 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7xt\" (UniqueName: \"kubernetes.io/projected/a3b4e81a-060a-4e6d-8b74-183f72a19b17-kube-api-access-4n7xt\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:00.953474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.953436 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:00.953474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.953449 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3b4e81a-060a-4e6d-8b74-183f72a19b17-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:00.997337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:00.997274 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:01.109333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.107498 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:01.333010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.321709 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:01.461943 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:45:01.461918 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8a125a_a607_4d5b_aab2_4e4742eef989.slice/crio-b20b412a903dd5800f0850d6b974a1dcb17dd6178e092d6bd72cb322d2c6e134 WatchSource:0}: Error finding container b20b412a903dd5800f0850d6b974a1dcb17dd6178e092d6bd72cb322d2c6e134: Status 404 returned error can't find the container with id b20b412a903dd5800f0850d6b974a1dcb17dd6178e092d6bd72cb322d2c6e134 Nov 03 17:45:01.464824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.464800 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:01.479410 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.479378 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" event={"ID":"4f8a125a-a607-4d5b-aab2-4e4742eef989","Type":"ContainerStarted","Data":"b20b412a903dd5800f0850d6b974a1dcb17dd6178e092d6bd72cb322d2c6e134"} Nov 03 17:45:01.480962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.480934 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerStarted","Data":"8feac19c317238f9c800ae7f62eb33f231183ccb53f9e04d3a6d7c6755e70356"} Nov 03 17:45:01.484579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.484554 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" Nov 03 17:45:01.484761 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.484579 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js" event={"ID":"a3b4e81a-060a-4e6d-8b74-183f72a19b17","Type":"ContainerDied","Data":"29d901b7c61d29477b43f6e5e1d893cc0922a761557c6be1498d65e0e0e95f1f"} Nov 03 17:45:01.484860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.484802 4885 scope.go:122] "RemoveContainer" containerID="22d314df936b0caaca200defab0ca12d9c46a2f3b3b85c44cdc30a2591fb8959" Nov 03 17:45:01.486350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.486333 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:01.486452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.486358 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" event={"ID":"4ba4dca5-ab42-4732-af9f-4c07627bd9d4","Type":"ContainerStarted","Data":"d934f73dc1db2b82bb76d23c9a497372589aef79ddca455d304306bc2b371ee0"} Nov 03 17:45:01.501420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.501398 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:01.571415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.571329 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\" (UniqueName: \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\") pod \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " Nov 03 17:45:01.571584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.571457 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\") pod \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\" (UID: \"48bfefa6-8a12-4c3c-af1e-e382c45afe5e\") " Nov 03 17:45:01.571789 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.571764 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca" pod "48bfefa6-8a12-4c3c-af1e-e382c45afe5e" (UID: "48bfefa6-8a12-4c3c-af1e-e382c45afe5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:01.602268 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.602219 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq" pod "48bfefa6-8a12-4c3c-af1e-e382c45afe5e" (UID: "48bfefa6-8a12-4c3c-af1e-e382c45afe5e"). InnerVolumeSpecName "kube-api-access-jpcwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:01.663914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.663886 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:45:01.664086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.663976 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0b55a36f-c45b655b4-vh6js"] Nov 03 17:45:01.673594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.673566 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:01.673725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:01.673596 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpcwq\" (UniqueName: \"kubernetes.io/projected/48bfefa6-8a12-4c3c-af1e-e382c45afe5e-kube-api-access-jpcwq\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:02.493163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.493101 4885 generic.go:356] "Generic (PLEG): container finished" podID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerID="a45c34c6b35ffd2e14f89fedd3673e02a6d7e76436f18f767ee0ce7649d898d6" exitCode=60 Nov 03 17:45:02.493547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.493167 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerDied","Data":"a45c34c6b35ffd2e14f89fedd3673e02a6d7e76436f18f767ee0ce7649d898d6"} Nov 03 17:45:02.493547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.493448 4885 scope.go:122] "RemoveContainer" containerID="a45c34c6b35ffd2e14f89fedd3673e02a6d7e76436f18f767ee0ce7649d898d6" Nov 03 17:45:02.496250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.496231 4885 generic.go:356] "Generic (PLEG): container finished" podID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerID="f4c110ebcf31b4a09bf6ef7c259a7f9d7b6d6fe883bc65dbcf64ff7b0db1e7bb" exitCode=60 Nov 03 17:45:02.496353 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.496335 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" event={"ID":"4ba4dca5-ab42-4732-af9f-4c07627bd9d4","Type":"ContainerDied","Data":"f4c110ebcf31b4a09bf6ef7c259a7f9d7b6d6fe883bc65dbcf64ff7b0db1e7bb"} Nov 03 17:45:02.496623 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.496539 4885 scope.go:122] "RemoveContainer" containerID="f4c110ebcf31b4a09bf6ef7c259a7f9d7b6d6fe883bc65dbcf64ff7b0db1e7bb" Nov 03 17:45:02.498432 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.498414 4885 generic.go:356] "Generic (PLEG): container finished" podID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerID="1d0668616d54b61288ff5fed966a4e502f258a9db5da7ec7c5a8753c08680f6f" exitCode=60 Nov 03 17:45:02.498532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.498458 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p" Nov 03 17:45:02.498532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.498495 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" event={"ID":"4f8a125a-a607-4d5b-aab2-4e4742eef989","Type":"ContainerDied","Data":"1d0668616d54b61288ff5fed966a4e502f258a9db5da7ec7c5a8753c08680f6f"} Nov 03 17:45:02.498826 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.498806 4885 scope.go:122] "RemoveContainer" containerID="1d0668616d54b61288ff5fed966a4e502f258a9db5da7ec7c5a8753c08680f6f" Nov 03 17:45:02.801031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.800999 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p"] Nov 03 17:45:02.811182 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:02.811159 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0b55a36f-29895465-pbm4p"] Nov 03 17:45:03.506616 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.506587 4885 generic.go:356] "Generic (PLEG): container finished" podID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerID="4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b" exitCode=60 Nov 03 17:45:03.507026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.506686 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" event={"ID":"4f8a125a-a607-4d5b-aab2-4e4742eef989","Type":"ContainerDied","Data":"4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b"} Nov 03 17:45:03.507026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.506772 4885 scope.go:122] "RemoveContainer" containerID="1d0668616d54b61288ff5fed966a4e502f258a9db5da7ec7c5a8753c08680f6f" Nov 03 17:45:03.507026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.507019 4885 scope.go:122] "RemoveContainer" containerID="4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b" Nov 03 17:45:03.507371 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:03.507351 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5_odh-ai-gateway-infra(4f8a125a-a607-4d5b-aab2-4e4742eef989)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" Nov 03 17:45:03.509475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.509450 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerStarted","Data":"bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b"} Nov 03 17:45:03.511289 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.511263 4885 generic.go:356] "Generic (PLEG): container finished" podID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerID="9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534" exitCode=60 Nov 03 17:45:03.511382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.511349 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" event={"ID":"4ba4dca5-ab42-4732-af9f-4c07627bd9d4","Type":"ContainerDied","Data":"9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534"} Nov 03 17:45:03.511558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.511541 4885 scope.go:122] "RemoveContainer" containerID="9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534" Nov 03 17:45:03.511833 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:03.511780 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m_odh-ai-gateway-infra(4ba4dca5-ab42-4732-af9f-4c07627bd9d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" Nov 03 17:45:03.562456 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.562435 4885 scope.go:122] "RemoveContainer" containerID="f4c110ebcf31b4a09bf6ef7c259a7f9d7b6d6fe883bc65dbcf64ff7b0db1e7bb" Nov 03 17:45:03.661749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.661684 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bfefa6-8a12-4c3c-af1e-e382c45afe5e" path="/var/lib/kubelet/pods/48bfefa6-8a12-4c3c-af1e-e382c45afe5e/volumes" Nov 03 17:45:03.661985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.661973 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" path="/var/lib/kubelet/pods/a3b4e81a-060a-4e6d-8b74-183f72a19b17/volumes" Nov 03 17:45:03.896946 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:03.896737 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" podStartSLOduration=3.896720598 podStartE2EDuration="3.896720598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:03.894218961 +0000 UTC m=+1646.348907997" watchObservedRunningTime="2026-11-03 17:45:03.896720598 +0000 UTC m=+1646.351409624" Nov 03 17:45:04.518810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.518773 4885 scope.go:122] "RemoveContainer" containerID="4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b" Nov 03 17:45:04.519286 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:04.519115 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5_odh-ai-gateway-infra(4f8a125a-a607-4d5b-aab2-4e4742eef989)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" Nov 03 17:45:04.520488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.520466 4885 generic.go:356] "Generic (PLEG): container finished" podID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerID="bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b" exitCode=60 Nov 03 17:45:04.520597 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.520552 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerDied","Data":"bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b"} Nov 03 17:45:04.520661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.520622 4885 scope.go:122] "RemoveContainer" containerID="a45c34c6b35ffd2e14f89fedd3673e02a6d7e76436f18f767ee0ce7649d898d6" Nov 03 17:45:04.520768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.520750 4885 scope.go:122] "RemoveContainer" containerID="bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b" Nov 03 17:45:04.521038 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:04.521018 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6_odh-ai-gateway-infra(3893ce97-f371-4c5c-94fb-bd497a067d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" Nov 03 17:45:04.522567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:04.522546 4885 scope.go:122] "RemoveContainer" containerID="9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534" Nov 03 17:45:04.522800 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:04.522781 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m_odh-ai-gateway-infra(4ba4dca5-ab42-4732-af9f-4c07627bd9d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" Nov 03 17:45:05.545658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:05.545628 4885 scope.go:122] "RemoveContainer" containerID="bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b" Nov 03 17:45:05.546132 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:05.545974 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6_odh-ai-gateway-infra(3893ce97-f371-4c5c-94fb-bd497a067d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" Nov 03 17:45:14.677092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:14.670116 4885 scope.go:122] "RemoveContainer" containerID="9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534" Nov 03 17:45:14.745963 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:14.745931 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:45:15.726113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:15.726010 4885 generic.go:356] "Generic (PLEG): container finished" podID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerID="623e96413d1be54935bf0799066b7df8c15ab05767535f93fcf662a0c93aae3a" exitCode=60 Nov 03 17:45:15.726113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:15.726089 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" event={"ID":"4ba4dca5-ab42-4732-af9f-4c07627bd9d4","Type":"ContainerDied","Data":"623e96413d1be54935bf0799066b7df8c15ab05767535f93fcf662a0c93aae3a"} Nov 03 17:45:15.726574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:15.726150 4885 scope.go:122] "RemoveContainer" containerID="9f4091b9cac44401d1dd14ee0b929090db37a09143d06d4ddcec8bd8ceedd534" Nov 03 17:45:15.726574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:15.726481 4885 scope.go:122] "RemoveContainer" containerID="623e96413d1be54935bf0799066b7df8c15ab05767535f93fcf662a0c93aae3a" Nov 03 17:45:15.726799 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:15.726778 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m_odh-ai-gateway-infra(4ba4dca5-ab42-4732-af9f-4c07627bd9d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" Nov 03 17:45:16.655358 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:16.655331 4885 scope.go:122] "RemoveContainer" containerID="4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b" Nov 03 17:45:16.875093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:16.873090 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:17.109769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.109744 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:17.137187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.137153 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\" (UniqueName: \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\") pod \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " Nov 03 17:45:17.137369 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.137351 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\") pod \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\" (UID: \"4ba4dca5-ab42-4732-af9f-4c07627bd9d4\") " Nov 03 17:45:17.137672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.137650 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca" pod "4ba4dca5-ab42-4732-af9f-4c07627bd9d4" (UID: "4ba4dca5-ab42-4732-af9f-4c07627bd9d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:17.137771 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.137756 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:17.139443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.139421 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4" pod "4ba4dca5-ab42-4732-af9f-4c07627bd9d4" (UID: "4ba4dca5-ab42-4732-af9f-4c07627bd9d4"). InnerVolumeSpecName "kube-api-access-gpnx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:17.238630 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.238596 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpnx4\" (UniqueName: \"kubernetes.io/projected/4ba4dca5-ab42-4732-af9f-4c07627bd9d4-kube-api-access-gpnx4\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:17.658858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.658831 4885 scope.go:122] "RemoveContainer" containerID="bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b" Nov 03 17:45:17.741984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.741962 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" Nov 03 17:45:17.742133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.742012 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m" event={"ID":"4ba4dca5-ab42-4732-af9f-4c07627bd9d4","Type":"ContainerDied","Data":"d934f73dc1db2b82bb76d23c9a497372589aef79ddca455d304306bc2b371ee0"} Nov 03 17:45:17.742133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.742096 4885 scope.go:122] "RemoveContainer" containerID="623e96413d1be54935bf0799066b7df8c15ab05767535f93fcf662a0c93aae3a" Nov 03 17:45:17.744197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.744169 4885 generic.go:356] "Generic (PLEG): container finished" podID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerID="8a7627979bcc6d6807cb93963995bf64fffde166c8a098153abbb10afa347e64" exitCode=60 Nov 03 17:45:17.744309 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.744294 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" event={"ID":"4f8a125a-a607-4d5b-aab2-4e4742eef989","Type":"ContainerDied","Data":"8a7627979bcc6d6807cb93963995bf64fffde166c8a098153abbb10afa347e64"} Nov 03 17:45:17.744672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.744655 4885 scope.go:122] "RemoveContainer" containerID="8a7627979bcc6d6807cb93963995bf64fffde166c8a098153abbb10afa347e64" Nov 03 17:45:17.744920 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:17.744899 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5_odh-ai-gateway-infra(4f8a125a-a607-4d5b-aab2-4e4742eef989)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" Nov 03 17:45:17.751091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:17.751042 4885 scope.go:122] "RemoveContainer" containerID="4ffe5a100b7aae92630a36f46e8586e2d5cfca356424e5c58182e0c609a5b68b" Nov 03 17:45:18.030103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.030047 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:18.037795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.037771 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-687fb3-29895465-zxk7m"] Nov 03 17:45:18.604178 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:18.604141 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:45:18.754901 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.754870 4885 generic.go:356] "Generic (PLEG): container finished" podID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerID="1ae15a802e72a1dd68fb0252af082426f28d9123338048270635309777c28d82" exitCode=60 Nov 03 17:45:18.755088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.754939 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerDied","Data":"1ae15a802e72a1dd68fb0252af082426f28d9123338048270635309777c28d82"} Nov 03 17:45:18.755088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.754997 4885 scope.go:122] "RemoveContainer" containerID="bd27cc100c979894b204fdeec39759fca13d6db4cf436306a2acdc4af8e0a58b" Nov 03 17:45:18.755365 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.755351 4885 scope.go:122] "RemoveContainer" containerID="1ae15a802e72a1dd68fb0252af082426f28d9123338048270635309777c28d82" Nov 03 17:45:18.755613 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:18.755591 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6_odh-ai-gateway-infra(3893ce97-f371-4c5c-94fb-bd497a067d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" Nov 03 17:45:18.925104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:18.925056 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:19.078798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.078779 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:19.159362 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.159321 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\") pod \"4f8a125a-a607-4d5b-aab2-4e4742eef989\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " Nov 03 17:45:19.159511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.159401 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\" (UniqueName: \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\") pod \"4f8a125a-a607-4d5b-aab2-4e4742eef989\" (UID: \"4f8a125a-a607-4d5b-aab2-4e4742eef989\") " Nov 03 17:45:19.159680 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.159659 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca" pod "4f8a125a-a607-4d5b-aab2-4e4742eef989" (UID: "4f8a125a-a607-4d5b-aab2-4e4742eef989"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:19.161540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.161516 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv" pod "4f8a125a-a607-4d5b-aab2-4e4742eef989" (UID: "4f8a125a-a607-4d5b-aab2-4e4742eef989"). InnerVolumeSpecName "kube-api-access-q6zsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:19.261233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.261201 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f8a125a-a607-4d5b-aab2-4e4742eef989-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:19.261233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.261224 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zsv\" (UniqueName: \"kubernetes.io/projected/4f8a125a-a607-4d5b-aab2-4e4742eef989-kube-api-access-q6zsv\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:19.662682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.662653 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" path="/var/lib/kubelet/pods/4ba4dca5-ab42-4732-af9f-4c07627bd9d4/volumes" Nov 03 17:45:19.766465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.766437 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" Nov 03 17:45:19.766624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.766432 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5" event={"ID":"4f8a125a-a607-4d5b-aab2-4e4742eef989","Type":"ContainerDied","Data":"b20b412a903dd5800f0850d6b974a1dcb17dd6178e092d6bd72cb322d2c6e134"} Nov 03 17:45:19.766624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.766602 4885 scope.go:122] "RemoveContainer" containerID="8a7627979bcc6d6807cb93963995bf64fffde166c8a098153abbb10afa347e64" Nov 03 17:45:19.883962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.883921 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:19.938408 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.938373 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:19.949135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:19.949103 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-147834-29895465-sjdn5"] Nov 03 17:45:20.045702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.045672 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:20.070660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.070638 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\" (UniqueName: \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\") pod \"3893ce97-f371-4c5c-94fb-bd497a067d64\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " Nov 03 17:45:20.070770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.070693 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\") pod \"3893ce97-f371-4c5c-94fb-bd497a067d64\" (UID: \"3893ce97-f371-4c5c-94fb-bd497a067d64\") " Nov 03 17:45:20.071037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.071013 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca" pod "3893ce97-f371-4c5c-94fb-bd497a067d64" (UID: "3893ce97-f371-4c5c-94fb-bd497a067d64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:20.072850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.072823 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d" pod "3893ce97-f371-4c5c-94fb-bd497a067d64" (UID: "3893ce97-f371-4c5c-94fb-bd497a067d64"). InnerVolumeSpecName "kube-api-access-dmz2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:20.172322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.172292 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmz2d\" (UniqueName: \"kubernetes.io/projected/3893ce97-f371-4c5c-94fb-bd497a067d64-kube-api-access-dmz2d\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:20.172322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.172316 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3893ce97-f371-4c5c-94fb-bd497a067d64-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:45:20.777877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.777836 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" event={"ID":"3893ce97-f371-4c5c-94fb-bd497a067d64","Type":"ContainerDied","Data":"8feac19c317238f9c800ae7f62eb33f231183ccb53f9e04d3a6d7c6755e70356"} Nov 03 17:45:20.778032 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.777915 4885 scope.go:122] "RemoveContainer" containerID="1ae15a802e72a1dd68fb0252af082426f28d9123338048270635309777c28d82" Nov 03 17:45:20.778032 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.778013 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6" Nov 03 17:45:20.951296 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.951263 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:20.958595 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:20.958557 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2355f4-29895465-clgg6"] Nov 03 17:45:21.662130 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:21.662100 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" path="/var/lib/kubelet/pods/3893ce97-f371-4c5c-94fb-bd497a067d64/volumes" Nov 03 17:45:21.662529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:21.662516 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" path="/var/lib/kubelet/pods/4f8a125a-a607-4d5b-aab2-4e4742eef989/volumes" Nov 03 17:45:38.150524 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:38.150491 4885 scope.go:122] "RemoveContainer" containerID="2d17dea77e288423cf2d3cbfce78021c6f62321dd71a6876cc8a301d0be15902" Nov 03 17:45:38.159286 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:38.159267 4885 scope.go:122] "RemoveContainer" containerID="c01b812936876d3a0238f94d47a271e994836c77627ca7d742efb14e0c8d7506" Nov 03 17:45:44.651562 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:44.651520 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:45:48.383140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383100 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383533 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383544 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383555 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383559 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383577 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383581 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383589 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383593 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383598 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.383596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383603 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383610 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383616 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383621 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383626 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383635 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383638 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383653 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" containerName="maas-api" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383657 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" containerName="maas-api" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383714 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383720 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383728 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383734 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383739 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383744 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383748 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3893ce97-f371-4c5c-94fb-bd497a067d64" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383754 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383758 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba4dca5-ab42-4732-af9f-4c07627bd9d4" containerName="cleanup" Nov 03 17:45:48.384030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.383764 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b4e81a-060a-4e6d-8b74-183f72a19b17" containerName="maas-api" Nov 03 17:45:48.386577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.386559 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.442930 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.442899 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:45:48.491208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.491168 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:45:48.491854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.491812 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3756d5e7-serving-cert\"" Nov 03 17:45:48.491854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.491834 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:45:48.491854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.491818 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:45:48.492133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.491815 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:45:48.542526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.542499 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.542680 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.542530 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.542722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.542698 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvfw\" (UniqueName: \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.542786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.542774 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.606998 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:45:48.606964 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:45:48.643933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.643840 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvfw\" (UniqueName: \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.643933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.643915 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.644171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.644011 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.644171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.644033 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.644697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.644677 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.646730 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.646699 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.646878 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.646845 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:48.707203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:48.707172 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvfw\" (UniqueName: \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\") pod \"maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:49.002733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:49.002655 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:49.377133 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:45:49.377110 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd669888_11b2_453d_8c2e_69f60c6aa79e.slice/crio-1f1f49304e95ff46a9d693ded59c5bf66f7b5617b2a5bb8b6bcec1d7a054d251 WatchSource:0}: Error finding container 1f1f49304e95ff46a9d693ded59c5bf66f7b5617b2a5bb8b6bcec1d7a054d251: Status 404 returned error can't find the container with id 1f1f49304e95ff46a9d693ded59c5bf66f7b5617b2a5bb8b6bcec1d7a054d251 Nov 03 17:45:49.380287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:49.380269 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:45:49.958338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:49.958298 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" event={"ID":"bd669888-11b2-453d-8c2e-69f60c6aa79e","Type":"ContainerStarted","Data":"1f1f49304e95ff46a9d693ded59c5bf66f7b5617b2a5bb8b6bcec1d7a054d251"} Nov 03 17:45:50.965678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:50.965643 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" event={"ID":"bd669888-11b2-453d-8c2e-69f60c6aa79e","Type":"ContainerStarted","Data":"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7"} Nov 03 17:45:50.966111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:50.965730 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:45:51.090738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:51.090678 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" podStartSLOduration=2.180500562 podStartE2EDuration="3.090659199s" totalImagesPullingTime="910.158637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:51.086515156 +0000 UTC m=+1693.541204182" watchObservedRunningTime="2026-11-03 17:45:51.090659199 +0000 UTC m=+1693.545348224" Nov 03 17:45:56.976390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:45:56.976361 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:46:13.969056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:13.969016 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:46:15.985670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:15.985640 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:46:15.986158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:15.986144 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:46:15.986218 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:15.986160 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8a125a-a607-4d5b-aab2-4e4742eef989" containerName="cleanup" Nov 03 17:46:15.988824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:15.988809 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017522 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017578 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017699 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017718 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017750 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017768 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5xb\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.017939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.017912 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.018028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.018013 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.018103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.018088 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.103396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.103353 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:46:16.119276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119245 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119280 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5xb\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119303 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119353 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119379 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119428 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119479 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119553 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119577 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119717 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.119924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.119875 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.120052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.120033 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.120216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.120194 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.120352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.120329 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.122150 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.122129 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.122704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.122687 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.182377 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.182344 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.182507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.182398 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5xb\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.302509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.302416 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:16.731855 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:46:16.731825 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4aef24f_6e2e_4c78_bb99_7a2614ea900b.slice/crio-977e05f53ea81f9ba7d08feb87408945c87b7a8a7439745ea30299774b0814c9 WatchSource:0}: Error finding container 977e05f53ea81f9ba7d08feb87408945c87b7a8a7439745ea30299774b0814c9: Status 404 returned error can't find the container with id 977e05f53ea81f9ba7d08feb87408945c87b7a8a7439745ea30299774b0814c9 Nov 03 17:46:16.733850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.733807 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:46:16.734907 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.734872 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:16.735005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.734952 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:16.735005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:16.734993 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:17.155115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.155053 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" event={"ID":"f4aef24f-6e2e-4c78-bb99-7a2614ea900b","Type":"ContainerStarted","Data":"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d"} Nov 03 17:46:17.155558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.155170 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" event={"ID":"f4aef24f-6e2e-4c78-bb99-7a2614ea900b","Type":"ContainerStarted","Data":"977e05f53ea81f9ba7d08feb87408945c87b7a8a7439745ea30299774b0814c9"} Nov 03 17:46:17.305495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.305456 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:17.321305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.321242 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podStartSLOduration=2.321221069 podStartE2EDuration="2.321221069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:17.278968552 +0000 UTC m=+1719.733657575" watchObservedRunningTime="2026-11-03 17:46:17.321221069 +0000 UTC m=+1719.775910095" Nov 03 17:46:17.323093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.323042 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:17.323328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.323292 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:17.761399 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:17.761364 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:46:18.322323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:18.311442 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:18.322323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:18.311502 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:18.630879 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:46:18.630842 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:46:18.728120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:18.728056 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-8c9b0f_payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n_1b8c273e-6a53-418d-b0ec-a94c4c71eb3c/payload-processing/0.log" Nov 03 17:46:19.315094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:19.305400 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:19.315094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:19.305457 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:20.305454 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:20.305417 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:20.306035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:20.306004 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:21.309091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:21.308432 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:21.309091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:21.308490 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:22.326552 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:22.326519 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 17:46:22.327185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:22.327155 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 17:46:23.494862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:23.494828 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:23.495704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:23.495685 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:23.496364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:23.496329 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:46:28.990095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:28.990006 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerName="main" containerID="cri-o://389f760274f985988a6599d18ddf8ed28bbc6cda77d63a8beaebbef36d07cd5d" gracePeriod=45 Nov 03 17:46:29.237776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.237745 4885 generic.go:356] "Generic (PLEG): container finished" podID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerID="389f760274f985988a6599d18ddf8ed28bbc6cda77d63a8beaebbef36d07cd5d" exitCode=0 Nov 03 17:46:29.237935 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.237827 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" event={"ID":"f1dbf390-a775-4a9c-8501-af0ec64ff19d","Type":"ContainerDied","Data":"389f760274f985988a6599d18ddf8ed28bbc6cda77d63a8beaebbef36d07cd5d"} Nov 03 17:46:29.263170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.263148 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417578 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417692 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\" (UniqueName: \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417781 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417851 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417882 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.418808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.417996 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\") pod \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\" (UID: \"f1dbf390-a775-4a9c-8501-af0ec64ff19d\") " Nov 03 17:46:29.419382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.418958 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:29.419382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.419146 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:29.421341 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.421306 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:29.424801 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.424644 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:29.426886 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.426856 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:29.435129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.435093 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp" pod "f1dbf390-a775-4a9c-8501-af0ec64ff19d" (UID: "f1dbf390-a775-4a9c-8501-af0ec64ff19d"). InnerVolumeSpecName "kube-api-access-vjfhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:29.520144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520044 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjfhp\" (UniqueName: \"kubernetes.io/projected/f1dbf390-a775-4a9c-8501-af0ec64ff19d-kube-api-access-vjfhp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:29.520144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520112 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:29.520144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520125 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:29.520144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520137 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:29.520144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520149 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbf390-a775-4a9c-8501-af0ec64ff19d-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:29.520407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:29.520159 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbf390-a775-4a9c-8501-af0ec64ff19d-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:30.322818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:30.319315 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" event={"ID":"f1dbf390-a775-4a9c-8501-af0ec64ff19d","Type":"ContainerDied","Data":"5b533d914cfccd5c5cb6303dbe93f70afd288961d196e8824afc5bd17991ed91"} Nov 03 17:46:30.322818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:30.319375 4885 scope.go:122] "RemoveContainer" containerID="389f760274f985988a6599d18ddf8ed28bbc6cda77d63a8beaebbef36d07cd5d" Nov 03 17:46:30.322818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:30.319535 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm" Nov 03 17:46:30.509533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:30.509498 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:46:30.519310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:30.519280 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-49c123d1/e2e-lifecycle-model-49c123d1-kserve-6f794474b9-wtcbm"] Nov 03 17:46:31.053548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.053517 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:46:31.104745 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.104702 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:46:31.104971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.104946 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" containerID="cri-o://b880f99f8c0339e00194a454eaac7886f7085ec582027d2060585d4f9b047a54" gracePeriod=30 Nov 03 17:46:31.107273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.107244 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:46:31.107450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.107430 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" containerID="cri-o://837c82cfa3733b6fed63e1f20bff6f9de625a457d12bb053b35e834c3c172279" gracePeriod=30 Nov 03 17:46:31.156780 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.156744 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:46:31.157036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.156994 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" containerID="cri-o://e551b0abdf015f7f7ebd4e3e85f0be3b8ba454262d2e86dfee63cf9d5d11afb3" gracePeriod=30 Nov 03 17:46:31.328005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.327917 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" containerID="cri-o://ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb" gracePeriod=30 Nov 03 17:46:31.662132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:31.662100 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" path="/var/lib/kubelet/pods/f1dbf390-a775-4a9c-8501-af0ec64ff19d/volumes" Nov 03 17:46:33.355704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:33.355663 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:46:33.356125 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:33.355742 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:46:34.024456 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.024426 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:46:34.024633 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.024482 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:46:34.027136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.027102 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:46:34.027259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.027174 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:46:34.746216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.746182 4885 patch_prober.go:29] interesting pod/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:46:34.746590 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:34.746243 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.366990 4885 generic.go:356] "Generic (PLEG): container finished" podID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerID="e551b0abdf015f7f7ebd4e3e85f0be3b8ba454262d2e86dfee63cf9d5d11afb3" exitCode=0 Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.367117 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" event={"ID":"977dfe00-1104-48ed-8df6-2f90673fbf92","Type":"ContainerDied","Data":"e551b0abdf015f7f7ebd4e3e85f0be3b8ba454262d2e86dfee63cf9d5d11afb3"} Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.368657 4885 generic.go:356] "Generic (PLEG): container finished" podID="66f3ccaa-f412-464d-890f-8524efbf560d" containerID="b880f99f8c0339e00194a454eaac7886f7085ec582027d2060585d4f9b047a54" exitCode=0 Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.368725 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" event={"ID":"66f3ccaa-f412-464d-890f-8524efbf560d","Type":"ContainerDied","Data":"b880f99f8c0339e00194a454eaac7886f7085ec582027d2060585d4f9b047a54"} Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.370373 4885 generic.go:356] "Generic (PLEG): container finished" podID="657602cf-f855-45bc-89f4-2af0ec76e834" containerID="837c82cfa3733b6fed63e1f20bff6f9de625a457d12bb053b35e834c3c172279" exitCode=0 Nov 03 17:46:36.373102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.370433 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" event={"ID":"657602cf-f855-45bc-89f4-2af0ec76e834","Type":"ContainerDied","Data":"837c82cfa3733b6fed63e1f20bff6f9de625a457d12bb053b35e834c3c172279"} Nov 03 17:46:36.483387 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.483363 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:46:36.488193 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.488112 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:46:36.538316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.538244 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541463 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541508 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541549 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541591 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541635 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541673 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541716 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541744 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541780 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541890 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541933 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541960 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.541994 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542053 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542171 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542208 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\") pod \"66f3ccaa-f412-464d-890f-8524efbf560d\" (UID: \"66f3ccaa-f412-464d-890f-8524efbf560d\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542248 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.542461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542273 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\") pod \"657602cf-f855-45bc-89f4-2af0ec76e834\" (UID: \"657602cf-f855-45bc-89f4-2af0ec76e834\") " Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542590 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542817 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542948 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.542962 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.543001 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657602cf-f855-45bc-89f4-2af0ec76e834-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.543176 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.543220 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.543459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.543393 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.543843 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.543651 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.544659 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.544411 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:36.545112 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.545080 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.545316 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.545277 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.548682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.548653 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "kube-api-access-p6ff6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.548791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.548748 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.549246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.549095 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:46:36.549246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.549143 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.549246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.549222 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.549575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.549541 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "kube-api-access-4thhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.550031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.549996 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy" pod "657602cf-f855-45bc-89f4-2af0ec76e834" (UID: "657602cf-f855-45bc-89f4-2af0ec76e834"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.551017 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.550998 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo" pod "66f3ccaa-f412-464d-890f-8524efbf560d" (UID: "66f3ccaa-f412-464d-890f-8524efbf560d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:46:36.643844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.643817 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.643850 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.643882 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.643920 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.643988 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644125 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644198 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.644241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644212 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644333 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644389 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\") pod \"977dfe00-1104-48ed-8df6-2f90673fbf92\" (UID: \"977dfe00-1104-48ed-8df6-2f90673fbf92\") " Nov 03 17:46:36.644760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644220 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.644760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644715 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.644882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644864 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.644938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644889 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.644938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644904 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.644938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644919 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6ff6\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-kube-api-access-p6ff6\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.644938 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644934 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f3ccaa-f412-464d-890f-8524efbf560d-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644947 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644961 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644971 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.644976 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657602cf-f855-45bc-89f4-2af0ec76e834-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645007 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657602cf-f855-45bc-89f4-2af0ec76e834-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645022 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f3ccaa-f412-464d-890f-8524efbf560d-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645036 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645050 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645101 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645116 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645130 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645144 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thhg\" (UniqueName: \"kubernetes.io/projected/66f3ccaa-f412-464d-890f-8524efbf560d-kube-api-access-4thhg\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645160 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f3ccaa-f412-464d-890f-8524efbf560d-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645177 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645191 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657602cf-f855-45bc-89f4-2af0ec76e834-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645244 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.645439 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.645381 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:46:36.646792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.646768 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:46:36.646905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.646793 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "kube-api-access-k9xrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.647283 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.647267 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.648543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.648523 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy" pod "977dfe00-1104-48ed-8df6-2f90673fbf92" (UID: "977dfe00-1104-48ed-8df6-2f90673fbf92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.745852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.745818 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.745852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.745856 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.745889 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.745920 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.745993 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746033 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746139 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746163 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.746299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746219 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.746299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746261 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746285 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.746545 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746306 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\") pod \"10776004-6223-4825-8ac7-edf61a79a43b\" (UID: \"10776004-6223-4825-8ac7-edf61a79a43b\") " Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746685 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746801 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10776004-6223-4825-8ac7-edf61a79a43b-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746819 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746883 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746920 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746936 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.746949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746949 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xrl\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-kube-api-access-k9xrl\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.747566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746962 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.747566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746975 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977dfe00-1104-48ed-8df6-2f90673fbf92-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.747566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.746988 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.747566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.747001 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.747566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.747014 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977dfe00-1104-48ed-8df6-2f90673fbf92-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.748818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.748796 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.748917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.748855 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:46:36.748972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.748920 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:36.748972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.748921 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6" pod "10776004-6223-4825-8ac7-edf61a79a43b" (UID: "10776004-6223-4825-8ac7-edf61a79a43b"). InnerVolumeSpecName "kube-api-access-58cv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:36.847851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.847815 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.847851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.847839 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.847851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.847848 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10776004-6223-4825-8ac7-edf61a79a43b-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.847851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.847857 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cv6\" (UniqueName: \"kubernetes.io/projected/10776004-6223-4825-8ac7-edf61a79a43b-kube-api-access-58cv6\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:36.848229 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:36.847865 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10776004-6223-4825-8ac7-edf61a79a43b-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:37.109098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.108905 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:46:37.376866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.376841 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" Nov 03 17:46:37.377318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.376842 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2" event={"ID":"66f3ccaa-f412-464d-890f-8524efbf560d","Type":"ContainerDied","Data":"7e1710b4980c811b925dcec5c57e3e704b829b829e75f8d25b5b6c4095b89f13"} Nov 03 17:46:37.377318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.376984 4885 scope.go:122] "RemoveContainer" containerID="b880f99f8c0339e00194a454eaac7886f7085ec582027d2060585d4f9b047a54" Nov 03 17:46:37.378406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.378387 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" Nov 03 17:46:37.378470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.378429 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr" event={"ID":"657602cf-f855-45bc-89f4-2af0ec76e834","Type":"ContainerDied","Data":"3bceba8cbcb018964a0bbc6a868e468007fb8cf275009c5d8bbafadd1be9b258"} Nov 03 17:46:37.380099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.380056 4885 generic.go:356] "Generic (PLEG): container finished" podID="10776004-6223-4825-8ac7-edf61a79a43b" containerID="ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb" exitCode=0 Nov 03 17:46:37.380203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.380130 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" Nov 03 17:46:37.380203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.380174 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" event={"ID":"10776004-6223-4825-8ac7-edf61a79a43b","Type":"ContainerDied","Data":"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb"} Nov 03 17:46:37.380304 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.380225 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr" event={"ID":"10776004-6223-4825-8ac7-edf61a79a43b","Type":"ContainerDied","Data":"01ed7060a484b6dc96bdf967b4b8538f4c7225f7c2eccf7d8a1c7b2233d9b940"} Nov 03 17:46:37.382664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.382643 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" event={"ID":"977dfe00-1104-48ed-8df6-2f90673fbf92","Type":"ContainerDied","Data":"c8180b466f5b7f541f62ef38af24bce8d612c687035231a6e801f99d119fdd07"} Nov 03 17:46:37.382743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.382700 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r" Nov 03 17:46:37.389011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.388216 4885 scope.go:122] "RemoveContainer" containerID="837c82cfa3733b6fed63e1f20bff6f9de625a457d12bb053b35e834c3c172279" Nov 03 17:46:37.398025 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.398009 4885 scope.go:122] "RemoveContainer" containerID="ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb" Nov 03 17:46:37.409763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.409744 4885 scope.go:122] "RemoveContainer" containerID="ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb" Nov 03 17:46:37.410530 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:46:37.410497 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb\": container with ID starting with ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb not found: ID does not exist" containerID="ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb" Nov 03 17:46:37.410628 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.410541 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb"} err="failed to get container status \"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb\": rpc error: code = NotFound desc = could not find container \"ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb\": container with ID starting with ed517cb75933b24726276e34dab11d4f0047198e7d6002bc7e5fb4297b163ddb not found: ID does not exist" Nov 03 17:46:37.410628 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.410566 4885 scope.go:122] "RemoveContainer" containerID="e551b0abdf015f7f7ebd4e3e85f0be3b8ba454262d2e86dfee63cf9d5d11afb3" Nov 03 17:46:37.547848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.547817 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:46:37.555489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.555465 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-tcg5r"] Nov 03 17:46:37.662543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.662470 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" path="/var/lib/kubelet/pods/977dfe00-1104-48ed-8df6-2f90673fbf92/volumes" Nov 03 17:46:37.677096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.677059 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:46:37.686204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.686178 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-cn8r2"] Nov 03 17:46:37.821446 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.821417 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:46:37.828084 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.828048 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-w6pbr"] Nov 03 17:46:37.943029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.942957 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:46:37.952109 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:37.952059 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-687fb3-openshift-default-67f5489557-nl5rr"] Nov 03 17:46:39.662971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:39.662941 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10776004-6223-4825-8ac7-edf61a79a43b" path="/var/lib/kubelet/pods/10776004-6223-4825-8ac7-edf61a79a43b/volumes" Nov 03 17:46:39.663370 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:39.663356 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" path="/var/lib/kubelet/pods/657602cf-f855-45bc-89f4-2af0ec76e834/volumes" Nov 03 17:46:39.663733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:39.663721 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" path="/var/lib/kubelet/pods/66f3ccaa-f412-464d-890f-8524efbf560d/volumes" Nov 03 17:46:42.110420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.110367 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" podUID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" containerName="payload-processing" containerID="cri-o://32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70" gracePeriod=25 Nov 03 17:46:42.361420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.361357 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:46:42.409827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.409800 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\") pod \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " Nov 03 17:46:42.409995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.409916 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\") pod \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " Nov 03 17:46:42.409995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.409959 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\") pod \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " Nov 03 17:46:42.409995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.409984 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\") pod \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\" (UID: \"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c\") " Nov 03 17:46:42.410428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.410399 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume" pod "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" (UID: "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:42.410542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.410438 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume" pod "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" (UID: "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:42.410542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.410519 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:42.412772 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.412747 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials" pod "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" (UID: "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:42.412884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.412854 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn" pod "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" (UID: "1b8c273e-6a53-418d-b0ec-a94c4c71eb3c"). InnerVolumeSpecName "kube-api-access-xgggn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:42.418511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.418486 4885 generic.go:356] "Generic (PLEG): container finished" podID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" containerID="32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70" exitCode=0 Nov 03 17:46:42.418607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.418541 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" Nov 03 17:46:42.418607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.418569 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" event={"ID":"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c","Type":"ContainerDied","Data":"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70"} Nov 03 17:46:42.418734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.418637 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n" event={"ID":"1b8c273e-6a53-418d-b0ec-a94c4c71eb3c","Type":"ContainerDied","Data":"514e0645ae1126cfc115aab4095551059267aef37b966bb32329eea8081bb142"} Nov 03 17:46:42.418734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.418674 4885 scope.go:122] "RemoveContainer" containerID="32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70" Nov 03 17:46:42.433222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.433193 4885 scope.go:122] "RemoveContainer" containerID="32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70" Nov 03 17:46:42.433478 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:46:42.433459 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70\": container with ID starting with 32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70 not found: ID does not exist" containerID="32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70" Nov 03 17:46:42.433556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.433482 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70"} err="failed to get container status \"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70\": rpc error: code = NotFound desc = could not find container \"32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70\": container with ID starting with 32176f7333bbeafdd6b207a288c144897026b3105941c870814281f4ec04ea70 not found: ID does not exist" Nov 03 17:46:42.511792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.511762 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgggn\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-kube-api-access-xgggn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:42.511792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.511791 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:42.511973 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.511804 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:42.605311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.605277 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:46:42.605504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:42.605342 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-6b887fm5n"] Nov 03 17:46:43.662186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:43.662153 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" path="/var/lib/kubelet/pods/1b8c273e-6a53-418d-b0ec-a94c4c71eb3c/volumes" Nov 03 17:46:48.606415 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:46:48.606375 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:46:51.126260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:46:51.126234 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:47:00.054853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:00.054818 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8"] Nov 03 17:47:00.107049 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:00.107013 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-685e356a-2ldp8"] Nov 03 17:47:01.662158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:01.662127 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81355d5a-7d30-411b-bf81-8a600742400e" path="/var/lib/kubelet/pods/81355d5a-7d30-411b-bf81-8a600742400e/volumes" Nov 03 17:47:11.053465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:11.053432 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb"] Nov 03 17:47:11.106120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:11.106036 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfac8996-k4tcb"] Nov 03 17:47:11.663503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:11.663467 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f" path="/var/lib/kubelet/pods/cfb4aab6-ede3-4ce7-8435-9255f6f7ff0f/volumes" Nov 03 17:47:12.598261 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:12.598231 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:47:12.598747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:12.598537 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" podUID="bd669888-11b2-453d-8c2e-69f60c6aa79e" containerName="maas-api" containerID="cri-o://f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7" gracePeriod=30 Nov 03 17:47:13.023102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.020783 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:47:13.109607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.109564 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\") pod \"bd669888-11b2-453d-8c2e-69f60c6aa79e\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " Nov 03 17:47:13.110138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.110116 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\") pod \"bd669888-11b2-453d-8c2e-69f60c6aa79e\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " Nov 03 17:47:13.110777 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.110756 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\") pod \"bd669888-11b2-453d-8c2e-69f60c6aa79e\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " Nov 03 17:47:13.111036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.111020 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\" (UniqueName: \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\") pod \"bd669888-11b2-453d-8c2e-69f60c6aa79e\" (UID: \"bd669888-11b2-453d-8c2e-69f60c6aa79e\") " Nov 03 17:47:13.111345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.109967 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca" pod "bd669888-11b2-453d-8c2e-69f60c6aa79e" (UID: "bd669888-11b2-453d-8c2e-69f60c6aa79e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:47:13.111743 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.111700 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd669888-11b2-453d-8c2e-69f60c6aa79e-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:13.114486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.113566 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw" pod "bd669888-11b2-453d-8c2e-69f60c6aa79e" (UID: "bd669888-11b2-453d-8c2e-69f60c6aa79e"). InnerVolumeSpecName "kube-api-access-qdvfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:47:13.115363 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.115327 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs" pod "bd669888-11b2-453d-8c2e-69f60c6aa79e" (UID: "bd669888-11b2-453d-8c2e-69f60c6aa79e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:47:13.115610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.115587 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls" pod "bd669888-11b2-453d-8c2e-69f60c6aa79e" (UID: "bd669888-11b2-453d-8c2e-69f60c6aa79e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:47:13.212877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.212790 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:13.212877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.212822 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvfw\" (UniqueName: \"kubernetes.io/projected/bd669888-11b2-453d-8c2e-69f60c6aa79e-kube-api-access-qdvfw\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:13.212877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.212833 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd669888-11b2-453d-8c2e-69f60c6aa79e-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:13.676886 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.676858 4885 generic.go:356] "Generic (PLEG): container finished" podID="bd669888-11b2-453d-8c2e-69f60c6aa79e" containerID="f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7" exitCode=0 Nov 03 17:47:13.677356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.676899 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" Nov 03 17:47:13.677356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.676936 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" event={"ID":"bd669888-11b2-453d-8c2e-69f60c6aa79e","Type":"ContainerDied","Data":"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7"} Nov 03 17:47:13.677356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.676987 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq" event={"ID":"bd669888-11b2-453d-8c2e-69f60c6aa79e","Type":"ContainerDied","Data":"1f1f49304e95ff46a9d693ded59c5bf66f7b5617b2a5bb8b6bcec1d7a054d251"} Nov 03 17:47:13.677356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.677015 4885 scope.go:122] "RemoveContainer" containerID="f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7" Nov 03 17:47:13.687973 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.687953 4885 scope.go:122] "RemoveContainer" containerID="f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7" Nov 03 17:47:13.688266 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:13.688244 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7\": container with ID starting with f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7 not found: ID does not exist" containerID="f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7" Nov 03 17:47:13.688333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.688277 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7"} err="failed to get container status \"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7\": rpc error: code = NotFound desc = could not find container \"f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7\": container with ID starting with f3102e6331cb7d6af0cc8e2b0fbcf6489e447fd10d103f7c0c023bf7ad0c80f7 not found: ID does not exist" Nov 03 17:47:13.857494 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.857456 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:47:13.872756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:13.872723 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3756d5e7-687bd87c94-qqptq"] Nov 03 17:47:15.661695 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:15.661663 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd669888-11b2-453d-8c2e-69f60c6aa79e" path="/var/lib/kubelet/pods/bd669888-11b2-453d-8c2e-69f60c6aa79e/volumes" Nov 03 17:47:15.736052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:15.736011 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:47:17.557154 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557120 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557542 4885 cpu_manager.go:414] "containerMap: removing container" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557554 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557566 4885 cpu_manager.go:414] "containerMap: removing container" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerName="main" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557570 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerName="main" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557587 4885 cpu_manager.go:414] "containerMap: removing container" podUID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" containerName="payload-processing" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557593 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" containerName="payload-processing" Nov 03 17:47:17.557608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557611 4885 cpu_manager.go:414] "containerMap: removing container" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557616 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557624 4885 cpu_manager.go:414] "containerMap: removing container" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557629 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557634 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bd669888-11b2-453d-8c2e-69f60c6aa79e" containerName="maas-api" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557638 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd669888-11b2-453d-8c2e-69f60c6aa79e" containerName="maas-api" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557646 4885 cpu_manager.go:414] "containerMap: removing container" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557650 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557702 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="977dfe00-1104-48ed-8df6-2f90673fbf92" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557710 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="10776004-6223-4825-8ac7-edf61a79a43b" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557716 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd669888-11b2-453d-8c2e-69f60c6aa79e" containerName="maas-api" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557723 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="657602cf-f855-45bc-89f4-2af0ec76e834" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557729 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f3ccaa-f412-464d-890f-8524efbf560d" containerName="istio-proxy" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557734 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dbf390-a775-4a9c-8501-af0ec64ff19d" containerName="main" Nov 03 17:47:17.557818 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.557741 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8c273e-6a53-418d-b0ec-a94c4c71eb3c" containerName="payload-processing" Nov 03 17:47:17.562150 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.562130 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.622714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.622682 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1b3c7bad-openshift-default-dockercfg-t8hrk\"" Nov 03 17:47:17.672037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.671999 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:47:17.721139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721099 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc4bq\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721147 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721286 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721325 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721464 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721518 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721615 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721587 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.721648 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.721628 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832451 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832530 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832587 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832625 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832685 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832768 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832818 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc4bq\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832848 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.832940 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.833721 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.835615 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.836139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.836040 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.859128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.846978 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.859644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.859610 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.862108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.861540 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.866096 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.862300 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.912585 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.910959 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:17.912585 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:17.912532 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc4bq\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\") pod \"e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:18.175311 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.175273 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:18.605104 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:18.605003 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:47:18.615806 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:47:18.615778 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127f3405_3e94_4b95_8bd9_2717865220e3.slice/crio-617e19d65bffc6b7b04780cc80578dbe02664301d54f03a5ed3bf6cf6ad32f9b WatchSource:0}: Error finding container 617e19d65bffc6b7b04780cc80578dbe02664301d54f03a5ed3bf6cf6ad32f9b: Status 404 returned error can't find the container with id 617e19d65bffc6b7b04780cc80578dbe02664301d54f03a5ed3bf6cf6ad32f9b Nov 03 17:47:18.618405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.618370 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:18.618495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.618451 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:18.618532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.618490 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:18.618828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.618807 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:47:18.747916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.747868 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerStarted","Data":"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13"} Nov 03 17:47:18.748108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.747933 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerStarted","Data":"617e19d65bffc6b7b04780cc80578dbe02664301d54f03a5ed3bf6cf6ad32f9b"} Nov 03 17:47:18.873657 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:18.873606 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podStartSLOduration=1.87359229 podStartE2EDuration="1.87359229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:18.87169995 +0000 UTC m=+1781.326389009" watchObservedRunningTime="2026-11-03 17:47:18.87359229 +0000 UTC m=+1781.328281313" Nov 03 17:47:19.175855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:19.175775 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:19.181648 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:19.181623 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:19.752509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:19.752482 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:19.753431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:19.753412 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:47:31.514272 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:31.514230 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:47:31.514762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:31.514435 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" containerID="cri-o://3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879" gracePeriod=30 Nov 03 17:47:34.126817 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:34.126779 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:47:34.127206 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:34.126837 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:47:36.762254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.762227 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:47:36.795293 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795233 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795293 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795264 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795338 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795384 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795457 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795522 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795563 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795627 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795656 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\") pod \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\" (UID: \"9e36f1e2-8fb5-424d-842f-b3fa89072f0a\") " Nov 03 17:47:36.795868 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795707 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:47:36.795868 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795777 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:47:36.796001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.795975 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:47:36.796176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.796148 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:47:36.796421 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.796400 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:47:36.796835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.796809 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.796956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.796945 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.797041 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.797031 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.797143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.797131 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.797219 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.797209 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.798349 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.798325 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:47:36.798664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.798619 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:47:36.799004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.798978 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:47:36.799004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.798997 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn" pod "9e36f1e2-8fb5-424d-842f-b3fa89072f0a" (UID: "9e36f1e2-8fb5-424d-842f-b3fa89072f0a"). InnerVolumeSpecName "kube-api-access-vkbpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:47:36.898584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.898554 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.898584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.898580 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.898584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.898592 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbpn\" (UniqueName: \"kubernetes.io/projected/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-kube-api-access-vkbpn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.898814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.898602 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e36f1e2-8fb5-424d-842f-b3fa89072f0a-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:47:36.919990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.919953 4885 generic.go:356] "Generic (PLEG): container finished" podID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerID="3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879" exitCode=0 Nov 03 17:47:36.920181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.920017 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" Nov 03 17:47:36.920181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.920018 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" event={"ID":"9e36f1e2-8fb5-424d-842f-b3fa89072f0a","Type":"ContainerDied","Data":"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879"} Nov 03 17:47:36.920181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.920134 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg" event={"ID":"9e36f1e2-8fb5-424d-842f-b3fa89072f0a","Type":"ContainerDied","Data":"7a4d0fb15001cc172ae8debe11d9ef25804e38539c5db4d9a012f2e75a6c23a4"} Nov 03 17:47:36.920181 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.920159 4885 scope.go:122] "RemoveContainer" containerID="3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879" Nov 03 17:47:36.931461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.931443 4885 scope.go:122] "RemoveContainer" containerID="3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879" Nov 03 17:47:36.931778 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:36.931751 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879\": container with ID starting with 3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879 not found: ID does not exist" containerID="3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879" Nov 03 17:47:36.931847 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:36.931789 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879"} err="failed to get container status \"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879\": rpc error: code = NotFound desc = could not find container \"3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879\": container with ID starting with 3fcfcc674d09cefdb03cdb1f73c15425bc8e3283e470b3652a3f5fb681513879 not found: ID does not exist" Nov 03 17:47:37.055828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:37.055732 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:47:37.059275 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:37.059248 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-jsrbg"] Nov 03 17:47:37.663883 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:37.663852 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" path="/var/lib/kubelet/pods/9e36f1e2-8fb5-424d-842f-b3fa89072f0a/volumes" Nov 03 17:47:43.731291 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:43.731255 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:47:48.639702 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:48.639659 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:47:53.664007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.663978 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:47:53.664401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.664366 4885 cpu_manager.go:414] "containerMap: removing container" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" Nov 03 17:47:53.664401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.664376 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" Nov 03 17:47:53.664481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.664471 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e36f1e2-8fb5-424d-842f-b3fa89072f0a" containerName="istio-proxy" Nov 03 17:47:53.667856 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.667841 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.773172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.773123 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:47:53.773172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.773119 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:47:53.773172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.773141 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1b3c7bad-serving-cert\"" Nov 03 17:47:53.773482 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.773176 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:47:53.773482 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.773296 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:47:53.775502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.775481 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:47:53.789978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.789951 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.790169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.789988 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.790169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.790019 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.790169 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.790140 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8q2z\" (UniqueName: \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.891028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.890985 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.891233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.891048 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.891233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.891111 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.891233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.891170 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8q2z\" (UniqueName: \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.891798 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:53.891773 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-serving-cert: secret "maas-api-e2e-derive-1b3c7bad-serving-cert" not found Nov 03 17:47:53.891964 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:47:53.891861 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls podName:c333dc3c-81b7-462c-a902-671f933e4a10 nodeName:}" failed. No retries permitted until 2026-11-03 17:47:54.391839812 +0000 UTC m=+1816.846528817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls") pod "maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" (UID: "c333dc3c-81b7-462c-a902-671f933e4a10") : secret "maas-api-e2e-derive-1b3c7bad-serving-cert" not found Nov 03 17:47:53.893332 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.892195 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.955145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.952628 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:53.973113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:53.969787 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8q2z\" (UniqueName: \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:54.397238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:54.397206 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:54.399790 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:54.399767 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") pod \"maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:54.580408 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:54.580376 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:54.952898 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:47:54.952869 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc333dc3c_81b7_462c_a902_671f933e4a10.slice/crio-a704d5e4c4258f9c9484a4361a220857707fe9c7e551f1d4f4b1d02e864249a2 WatchSource:0}: Error finding container a704d5e4c4258f9c9484a4361a220857707fe9c7e551f1d4f4b1d02e864249a2: Status 404 returned error can't find the container with id a704d5e4c4258f9c9484a4361a220857707fe9c7e551f1d4f4b1d02e864249a2 Nov 03 17:47:54.954561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:54.954539 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:47:54.955404 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:54.955343 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:47:55.152388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:55.152355 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" event={"ID":"c333dc3c-81b7-462c-a902-671f933e4a10","Type":"ContainerStarted","Data":"a704d5e4c4258f9c9484a4361a220857707fe9c7e551f1d4f4b1d02e864249a2"} Nov 03 17:47:56.158733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:56.158693 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" event={"ID":"c333dc3c-81b7-462c-a902-671f933e4a10","Type":"ContainerStarted","Data":"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041"} Nov 03 17:47:56.159207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:56.158751 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:47:56.282206 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:56.282155 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" podStartSLOduration=2.1916453049999998 podStartE2EDuration="3.282141575s" totalImagesPullingTime="1.09049627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:56.27973212 +0000 UTC m=+1818.734421144" watchObservedRunningTime="2026-11-03 17:47:56.282141575 +0000 UTC m=+1818.736830598" Nov 03 17:47:59.869153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:59.854197 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:47:59.869153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:47:59.854302 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:00.836091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:00.835147 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:02.172035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:02.171998 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:48:05.250808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:05.250777 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:48:08.810220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:08.810172 4885 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:08.832496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:08.832442 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 17:48:08.989093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:08.982305 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.093149 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.085188 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.124095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.123209 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.419226 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.419171 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.461197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.461151 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.623123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.623087 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-xv8cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:48:09.623454 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.623420 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.884265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.884209 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:09.885457 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.885432 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:48:09.991252 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.991200 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Nov 03 17:48:09.992990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:09.992178 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:48:10.100095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.096380 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:10.100095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.097166 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:48:10.124423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.124155 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:10.235160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.210189 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:48:10.424242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.423454 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:10.434091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.424477 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:48:10.484481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.484440 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:10.485326 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.485262 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:48:10.901555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:10.885159 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 17:48:11.048091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.046655 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:11.105612 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.104345 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": context deadline exceeded" Nov 03 17:48:11.126186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.126143 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:11.201911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.201826 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:48:11.400822 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.400155 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:48:11.433208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.433163 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:11.686860 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.686810 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:48:11.970733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.970644 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:48:11.993702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:11.993672 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:48:12.014657 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:12.014629 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:48:16.476465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.476377 4885 generic.go:356] "Generic (PLEG): container finished" podID="127f3405-3e94-4b95-8bd9-2717865220e3" containerID="0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13" exitCode=0 Nov 03 17:48:16.476465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.476444 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerDied","Data":"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13"} Nov 03 17:48:16.476918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.476827 4885 scope.go:122] "RemoveContainer" containerID="0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13" Nov 03 17:48:16.478124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.478101 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:16.478179 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.478166 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:16.478212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:16.478200 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:17.484460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:17.484417 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerStarted","Data":"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262"} Nov 03 17:48:18.175857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:18.175824 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:48:18.177042 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:18.177016 4885 patch_prober.go:29] interesting pod/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 17:48:18.177197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:18.177099 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 17:48:18.642218 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:48:18.642180 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:48:19.176687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:19.176642 4885 patch_prober.go:29] interesting pod/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 17:48:19.176890 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:19.176692 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 17:48:19.752457 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:19.752423 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:48:20.175844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:20.175808 4885 patch_prober.go:29] interesting pod/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 17:48:20.176031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:20.175862 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 17:48:21.175644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:21.175615 4885 patch_prober.go:29] interesting pod/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 17:48:21.176012 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:21.175663 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 17:48:22.176047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:22.175846 4885 patch_prober.go:29] interesting pod/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 17:48:22.176047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:22.175896 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 17:48:23.180790 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:23.180761 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:48:23.181678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:23.181659 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:48:31.990982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:31.990948 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:48:31.991476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:31.991173 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" containerID="cri-o://6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f" gracePeriod=30 Nov 03 17:48:34.451767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:34.451736 4885 patch_prober.go:29] interesting pod/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:48:34.452163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:34.451795 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:48:37.306106 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.306052 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:48:37.327460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327429 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327526 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327694 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327664 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327691 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:48:37.327753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327732 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327752 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327774 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.327854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327824 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.328001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327852 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.328001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327899 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:48:37.328001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.327934 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\") pod \"9f474b0f-5891-42e2-8bff-d7b3217aa547\" (UID: \"9f474b0f-5891-42e2-8bff-d7b3217aa547\") " Nov 03 17:48:37.328174 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328150 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:48:37.328254 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328227 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:37.328375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328347 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:48:37.328668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328406 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f474b0f-5891-42e2-8bff-d7b3217aa547-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.328668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328431 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.328668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328449 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.328668 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.328463 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.330133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.330112 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:37.330346 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.330324 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:48:37.330563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.330534 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "kube-api-access-47fsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:37.330658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.330605 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo" pod "9f474b0f-5891-42e2-8bff-d7b3217aa547" (UID: "9f474b0f-5891-42e2-8bff-d7b3217aa547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:48:37.429663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.429565 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fsp\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-kube-api-access-47fsp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.429663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.429598 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.429663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.429613 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.429663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.429626 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f474b0f-5891-42e2-8bff-d7b3217aa547-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.429663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.429639 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f474b0f-5891-42e2-8bff-d7b3217aa547-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:37.657183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.657149 4885 generic.go:356] "Generic (PLEG): container finished" podID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerID="6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f" exitCode=0 Nov 03 17:48:37.658467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.658445 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" Nov 03 17:48:37.662887 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.662863 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" event={"ID":"9f474b0f-5891-42e2-8bff-d7b3217aa547","Type":"ContainerDied","Data":"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f"} Nov 03 17:48:37.662992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.662905 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8" event={"ID":"9f474b0f-5891-42e2-8bff-d7b3217aa547","Type":"ContainerDied","Data":"48e3f14e4471f26093da43db4bd701cc0e7e8f1c732cd8d1b7a8c28048d48b71"} Nov 03 17:48:37.662992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.662927 4885 scope.go:122] "RemoveContainer" containerID="6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f" Nov 03 17:48:37.676625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.676406 4885 scope.go:122] "RemoveContainer" containerID="6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f" Nov 03 17:48:37.677116 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:48:37.677086 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f\": container with ID starting with 6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f not found: ID does not exist" containerID="6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f" Nov 03 17:48:37.677212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.677128 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f"} err="failed to get container status \"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f\": rpc error: code = NotFound desc = could not find container \"6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f\": container with ID starting with 6093f857c49bf0b8a4d72a6eeeb11aa21d5c25e1c685917beae7706985341a4f not found: ID does not exist" Nov 03 17:48:37.844083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.844024 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:48:37.845763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:37.845741 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-mhcb8"] Nov 03 17:48:38.254334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:38.254303 4885 scope.go:122] "RemoveContainer" containerID="a1f009f9bb711ba1a1ad73dff4b4bd4066b972fee23482195cce8e4fe8e3c49b" Nov 03 17:48:38.262953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:38.262933 4885 scope.go:122] "RemoveContainer" containerID="86be1b0a1c01c207007b996916d3c73a921b0889b2274215472b84f403f59da7" Nov 03 17:48:38.270894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:38.270874 4885 scope.go:122] "RemoveContainer" containerID="33612f25920eb098a871f0fcb421c5aa3ceb5d5647b9adbac9ce770455857187" Nov 03 17:48:39.661953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:39.661918 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" path="/var/lib/kubelet/pods/9f474b0f-5891-42e2-8bff-d7b3217aa547/volumes" Nov 03 17:48:40.341507 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:40.341479 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:48:42.683769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.683737 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz"] Nov 03 17:48:42.685296 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.685266 4885 cpu_manager.go:414] "containerMap: removing container" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" Nov 03 17:48:42.685460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.685444 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" Nov 03 17:48:42.685748 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.685733 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f474b0f-5891-42e2-8bff-d7b3217aa547" containerName="istio-proxy" Nov 03 17:48:42.689834 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.689817 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.724168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.724131 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdp2\" (UniqueName: \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.724330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.724198 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.724330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.724253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.744237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.744204 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:48:42.796011 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.795970 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:48:42.798993 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.798950 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz"] Nov 03 17:48:42.868132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.848593 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdp2\" (UniqueName: \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.868132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.848679 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.868132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.848748 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.868132 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.849836 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.869699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.869671 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:42.939144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:42.934137 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdp2\" (UniqueName: \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\") pod \"maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:43.020255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:43.020223 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:44.076158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.073810 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz"] Nov 03 17:48:44.436092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.433141 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:44.845116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.845006 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" event={"ID":"c98c7165-48c0-4dd8-98b0-a8dcf1da5264","Type":"ContainerStarted","Data":"c277a5488115e05cbac651b756c5e5e110c4b1956e9c87014727c1725c1c4773"} Nov 03 17:48:44.845372 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.845352 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" event={"ID":"c98c7165-48c0-4dd8-98b0-a8dcf1da5264","Type":"ContainerStarted","Data":"4b3cc8d6702110a38212c61cde4da15739d0c9a8daca6973de140f37ebb109fe"} Nov 03 17:48:44.872205 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.872163 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:48:44.872413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:44.872227 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:48:45.495094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:45.492158 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" podStartSLOduration=3.492137536 podStartE2EDuration="3.492137536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:45.015353237 +0000 UTC m=+1867.470042262" watchObservedRunningTime="2026-11-03 17:48:45.492137536 +0000 UTC m=+1867.946826561" Nov 03 17:48:45.495718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:45.495693 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:48:45.529327 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:45.519476 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:48:45.572977 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:45.545588 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:48:46.040356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:46.039800 4885 generic.go:356] "Generic (PLEG): container finished" podID="c98c7165-48c0-4dd8-98b0-a8dcf1da5264" containerID="c277a5488115e05cbac651b756c5e5e110c4b1956e9c87014727c1725c1c4773" exitCode=0 Nov 03 17:48:46.040356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:46.039967 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" event={"ID":"c98c7165-48c0-4dd8-98b0-a8dcf1da5264","Type":"ContainerDied","Data":"c277a5488115e05cbac651b756c5e5e110c4b1956e9c87014727c1725c1c4773"} Nov 03 17:48:47.171761 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.171732 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:48:47.175700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.175675 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.197764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.197742 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:47.262644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.262605 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:48:47.335894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.335814 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\" (UniqueName: \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\") pod \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " Nov 03 17:48:47.335894 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.335888 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\") pod \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " Nov 03 17:48:47.336111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336039 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\") pod \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\" (UID: \"c98c7165-48c0-4dd8-98b0-a8dcf1da5264\") " Nov 03 17:48:47.336335 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336311 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336348 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336425 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca" pod "c98c7165-48c0-4dd8-98b0-a8dcf1da5264" (UID: "c98c7165-48c0-4dd8-98b0-a8dcf1da5264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:47.336542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336455 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336482 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336542 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336505 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336627 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcrv\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.336913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336892 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.337016 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.336999 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.337053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.337043 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.337253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.337234 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:47.338253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.338227 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca" pod "c98c7165-48c0-4dd8-98b0-a8dcf1da5264" (UID: "c98c7165-48c0-4dd8-98b0-a8dcf1da5264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:48:47.338564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.338544 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2" pod "c98c7165-48c0-4dd8-98b0-a8dcf1da5264" (UID: "c98c7165-48c0-4dd8-98b0-a8dcf1da5264"). InnerVolumeSpecName "kube-api-access-gzdp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.438967 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439059 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439132 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439199 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439225 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439259 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439280 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439308 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439354 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcrv\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439422 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzdp2\" (UniqueName: \"kubernetes.io/projected/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-kube-api-access-gzdp2\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439440 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c98c7165-48c0-4dd8-98b0-a8dcf1da5264-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:48:47.439911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.439491 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.441395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.440920 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.441395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.441002 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.441607 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.441582 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.443234 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.443211 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.447359 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.447333 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.447517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.447498 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.502172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.502137 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcrv\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.504949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.504916 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\") pod \"e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.509978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.509942 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:47.943139 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.942815 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:48:47.947633 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:48:47.946265 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eabb1a3_fcc8_47c0_bb9d_133c8844d483.slice/crio-7530b5c433859dffe86619291e88dc4a39f8825c22c6670442bf6e25372a16fc WatchSource:0}: Error finding container 7530b5c433859dffe86619291e88dc4a39f8825c22c6670442bf6e25372a16fc: Status 404 returned error can't find the container with id 7530b5c433859dffe86619291e88dc4a39f8825c22c6670442bf6e25372a16fc Nov 03 17:48:47.952370 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.952329 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:47.952509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.952422 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:47.952509 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:47.952461 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:48.054515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.054466 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" event={"ID":"7eabb1a3-fcc8-47c0-bb9d-133c8844d483","Type":"ContainerStarted","Data":"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff"} Nov 03 17:48:48.054690 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.054554 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" event={"ID":"7eabb1a3-fcc8-47c0-bb9d-133c8844d483","Type":"ContainerStarted","Data":"7530b5c433859dffe86619291e88dc4a39f8825c22c6670442bf6e25372a16fc"} Nov 03 17:48:48.056055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.056030 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" event={"ID":"c98c7165-48c0-4dd8-98b0-a8dcf1da5264","Type":"ContainerDied","Data":"4b3cc8d6702110a38212c61cde4da15739d0c9a8daca6973de140f37ebb109fe"} Nov 03 17:48:48.056055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.056084 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3cc8d6702110a38212c61cde4da15739d0c9a8daca6973de140f37ebb109fe" Nov 03 17:48:48.056242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.056101 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz" Nov 03 17:48:48.179810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.179749 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podStartSLOduration=1.1797286009999999 podStartE2EDuration="1.179728601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:48.177577387 +0000 UTC m=+1870.632266421" watchObservedRunningTime="2026-11-03 17:48:48.179728601 +0000 UTC m=+1870.634417623" Nov 03 17:48:48.515198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:48.515163 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:48.630270 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:48:48.630230 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:48:49.520094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:49.517306 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 17:48:49.520094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:49.517367 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 17:48:50.510490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:50.510450 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 17:48:50.510891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:50.510858 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 17:48:51.514449 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:51.514407 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 17:48:51.515086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:51.515023 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 17:48:52.124092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.122901 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:48:52.124092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.123586 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c98c7165-48c0-4dd8-98b0-a8dcf1da5264" containerName="revoke-keys" Nov 03 17:48:52.124092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.123604 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98c7165-48c0-4dd8-98b0-a8dcf1da5264" containerName="revoke-keys" Nov 03 17:48:52.124092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.123716 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98c7165-48c0-4dd8-98b0-a8dcf1da5264" containerName="revoke-keys" Nov 03 17:48:52.130208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.130181 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.191799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.191770 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.191886 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.191933 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.191967 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.191987 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.192190 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.192271 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.192301 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95k4c\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.192461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.192327 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.241017 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.239964 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:48:52.292903 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.292875 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.293102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.292909 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.293102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.292943 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.292997 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.293611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.293426 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95k4c\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.293813 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.293869 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.293909 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.294013 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.294543 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.294119 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.297635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.295271 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.297635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.296154 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.297635 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.296337 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.336267 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.334667 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.336267 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.335611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.356258 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.356231 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.356441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.356416 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95k4c\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\") pod \"e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.444610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.444518 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:52.542631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.517470 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 17:48:52.542631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.517530 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 17:48:52.874017 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:48:52.873985 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892b780d_778e_4cb4_9140_6cf014dab4c1.slice/crio-a0c6ec777532d9a448ef84a1300f6379e9ce793f7cb543d5004276a2c00da6b5 WatchSource:0}: Error finding container a0c6ec777532d9a448ef84a1300f6379e9ce793f7cb543d5004276a2c00da6b5: Status 404 returned error can't find the container with id a0c6ec777532d9a448ef84a1300f6379e9ce793f7cb543d5004276a2c00da6b5 Nov 03 17:48:52.876644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.876615 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:52.876735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.876678 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:52.876735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.876696 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:48:52.876735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:52.876710 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:53.083510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.082980 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" event={"ID":"892b780d-778e-4cb4-9140-6cf014dab4c1","Type":"ContainerStarted","Data":"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941"} Nov 03 17:48:53.083510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.083040 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" event={"ID":"892b780d-778e-4cb4-9140-6cf014dab4c1","Type":"ContainerStarted","Data":"a0c6ec777532d9a448ef84a1300f6379e9ce793f7cb543d5004276a2c00da6b5"} Nov 03 17:48:53.220163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.220031 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podStartSLOduration=1.220011707 podStartE2EDuration="1.220011707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:53.219153681 +0000 UTC m=+1875.673842705" watchObservedRunningTime="2026-11-03 17:48:53.220011707 +0000 UTC m=+1875.674700731" Nov 03 17:48:53.446531 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.446491 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:53.447659 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.447635 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:53.447788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.447680 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:53.514497 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.514404 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 03 17:48:53.514497 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:53.514466 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 03 17:48:54.455279 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:54.455246 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:54.455821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:54.455791 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:55.025873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:55.024411 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:55.025873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:55.025728 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:55.048702 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:55.048665 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:48:55.445300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:55.445261 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:55.445495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:55.445316 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:56.445983 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:56.445945 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:56.446413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:56.446002 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:57.445587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:57.445549 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:57.445772 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:57.445608 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:58.445060 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:58.445022 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:48:58.445491 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:58.445105 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 17:48:59.729188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:59.729157 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:59.731115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:59.731096 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:48:59.732374 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:48:59.732322 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:49:00.687327 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:00.687285 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" containerID="cri-o://fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4" gracePeriod=45 Nov 03 17:49:00.756584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:00.756532 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" containerID="cri-o://a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4" gracePeriod=45 Nov 03 17:49:00.998620 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:00.998546 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" containerID="cri-o://a52db638911aec2a4ce2ddaaff69682a101e743bae64b4fa62fe8d0e4fb45c93" gracePeriod=45 Nov 03 17:49:01.152788 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.152281 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:49:01.154250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.154224 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:49:01.159370 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.159345 4885 generic.go:356] "Generic (PLEG): container finished" podID="58e5f376-8ed7-49b4-8779-12218aa50387" containerID="a52db638911aec2a4ce2ddaaff69682a101e743bae64b4fa62fe8d0e4fb45c93" exitCode=0 Nov 03 17:49:01.159500 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.159432 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" event={"ID":"58e5f376-8ed7-49b4-8779-12218aa50387","Type":"ContainerDied","Data":"a52db638911aec2a4ce2ddaaff69682a101e743bae64b4fa62fe8d0e4fb45c93"} Nov 03 17:49:01.160734 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.160709 4885 generic.go:356] "Generic (PLEG): container finished" podID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerID="a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4" exitCode=0 Nov 03 17:49:01.160855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.160743 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" Nov 03 17:49:01.160855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.160788 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" event={"ID":"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8","Type":"ContainerDied","Data":"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4"} Nov 03 17:49:01.160855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.160830 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" event={"ID":"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8","Type":"ContainerDied","Data":"07c9915680816b6c32633bf826056caaaf075371b89c5117c79f4e74627b34e1"} Nov 03 17:49:01.160998 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.160857 4885 scope.go:122] "RemoveContainer" containerID="a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4" Nov 03 17:49:01.162203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.162183 4885 generic.go:356] "Generic (PLEG): container finished" podID="6b796743-59cf-464d-90a7-b1b0226f9748" containerID="fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4" exitCode=0 Nov 03 17:49:01.162286 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.162228 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" Nov 03 17:49:01.162344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.162276 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" event={"ID":"6b796743-59cf-464d-90a7-b1b0226f9748","Type":"ContainerDied","Data":"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4"} Nov 03 17:49:01.162344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.162312 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99" event={"ID":"6b796743-59cf-464d-90a7-b1b0226f9748","Type":"ContainerDied","Data":"2d8aba4dada0f6d0e105923119ee96f364f09853ac85f72acbe47866c6715c85"} Nov 03 17:49:01.174511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.173520 4885 scope.go:122] "RemoveContainer" containerID="a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4" Nov 03 17:49:01.174877 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:01.174723 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4\": container with ID starting with a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4 not found: ID does not exist" containerID="a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4" Nov 03 17:49:01.174877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.174766 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4"} err="failed to get container status \"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4\": rpc error: code = NotFound desc = could not find container \"a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4\": container with ID starting with a82357846d9559c22c13bfc343465cb260c8552c9e7052f66bd1f2796960f4a4 not found: ID does not exist" Nov 03 17:49:01.174877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.174788 4885 scope.go:122] "RemoveContainer" containerID="fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4" Nov 03 17:49:01.186245 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.186217 4885 scope.go:122] "RemoveContainer" containerID="fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4" Nov 03 17:49:01.187906 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:01.186533 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4\": container with ID starting with fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4 not found: ID does not exist" containerID="fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4" Nov 03 17:49:01.187906 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.186568 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4"} err="failed to get container status \"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4\": rpc error: code = NotFound desc = could not find container \"fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4\": container with ID starting with fd0b8e05c8704add73d183f7798a89d46d7d3940e210bca0494c63961a63b9e4 not found: ID does not exist" Nov 03 17:49:01.275486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275456 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\" (UniqueName: \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.275662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275508 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.275662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275559 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.275662 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275596 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\" (UniqueName: \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.275774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275670 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.275774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275724 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.275774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275751 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.276005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275787 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.276005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275831 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.276005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275890 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.276005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275917 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\") pod \"6b796743-59cf-464d-90a7-b1b0226f9748\" (UID: \"6b796743-59cf-464d-90a7-b1b0226f9748\") " Nov 03 17:49:01.276005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.275998 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\") pod \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\" (UID: \"d3a02bf5-bb41-40a7-92fb-68acb2d5cca8\") " Nov 03 17:49:01.278188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.276738 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.278188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.277659 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.278188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.277882 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.278188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.277912 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.278188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.278130 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.280805 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.280190 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.280805 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.280278 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "kube-api-access-hbfgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:01.282124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.281041 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:01.285469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.285441 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.287001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.286979 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:01.287326 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.287303 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm" pod "6b796743-59cf-464d-90a7-b1b0226f9748" (UID: "6b796743-59cf-464d-90a7-b1b0226f9748"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.289105 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.289040 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k" pod "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" (UID: "d3a02bf5-bb41-40a7-92fb-68acb2d5cca8"). InnerVolumeSpecName "kube-api-access-z8m4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:01.302913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.302839 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:49:01.383814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.383776 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384003 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.383843 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384103 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\" (UniqueName: \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384187 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384239 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\" (UniqueName: \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384321 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384299 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.384419 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384386 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\") pod \"58e5f376-8ed7-49b4-8779-12218aa50387\" (UID: \"58e5f376-8ed7-49b4-8779-12218aa50387\") " Nov 03 17:49:01.384502 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384439 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.384601 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384582 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.384852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384833 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.384981 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385003 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385016 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385029 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b796743-59cf-464d-90a7-b1b0226f9748-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385040 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385052 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385096 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8m4k\" (UniqueName: \"kubernetes.io/projected/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-kube-api-access-z8m4k\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385108 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385117 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385119 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385131 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbfgp\" (UniqueName: \"kubernetes.io/projected/6b796743-59cf-464d-90a7-b1b0226f9748-kube-api-access-hbfgp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385143 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385154 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385165 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.385532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.385175 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b796743-59cf-464d-90a7-b1b0226f9748-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.386379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.386352 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "kube-api-access-89wwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:01.386462 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.386429 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:01.386462 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.386447 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs" pod "58e5f376-8ed7-49b4-8779-12218aa50387" (UID: "58e5f376-8ed7-49b4-8779-12218aa50387"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:01.485982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.485955 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58e5f376-8ed7-49b4-8779-12218aa50387-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.485982 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.485982 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89wwk\" (UniqueName: \"kubernetes.io/projected/58e5f376-8ed7-49b4-8779-12218aa50387-kube-api-access-89wwk\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.486174 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.485993 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58e5f376-8ed7-49b4-8779-12218aa50387-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:01.723284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.723145 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:49:01.724431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.724288 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-w0-687fb3-kserve-858b8df8f-k8h99"] Nov 03 17:49:01.847241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.845731 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:49:01.851414 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:01.851362 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv"] Nov 03 17:49:02.019767 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.019678 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-687fb3/premium-w0-687fb3-kserve-579c56df4b-vdjfv" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:49:02.168764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.168733 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" Nov 03 17:49:02.168764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.168746 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx" event={"ID":"58e5f376-8ed7-49b4-8779-12218aa50387","Type":"ContainerDied","Data":"00891ece491c692fd01e127cd960c12d96fdee7c40f0a632171a5a2355cae0ce"} Nov 03 17:49:02.169087 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.168806 4885 scope.go:122] "RemoveContainer" containerID="a52db638911aec2a4ce2ddaaff69682a101e743bae64b4fa62fe8d0e4fb45c93" Nov 03 17:49:02.359160 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.355391 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:49:02.379867 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:02.379812 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-687fb3/distinct-2-w0-687fb3-kserve-97898d7cd-2tlsx"] Nov 03 17:49:03.668516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:03.668461 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" path="/var/lib/kubelet/pods/58e5f376-8ed7-49b4-8779-12218aa50387/volumes" Nov 03 17:49:03.670900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:03.670869 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" path="/var/lib/kubelet/pods/6b796743-59cf-464d-90a7-b1b0226f9748/volumes" Nov 03 17:49:03.671506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:03.671484 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" path="/var/lib/kubelet/pods/d3a02bf5-bb41-40a7-92fb-68acb2d5cca8/volumes" Nov 03 17:49:11.450537 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.450504 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:49:11.451004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.450686 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" podUID="c333dc3c-81b7-462c-a902-671f933e4a10" containerName="maas-api" containerID="cri-o://efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041" gracePeriod=30 Nov 03 17:49:11.860790 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.860392 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:49:11.993932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.993885 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\") pod \"c333dc3c-81b7-462c-a902-671f933e4a10\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " Nov 03 17:49:11.994143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.994026 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") pod \"c333dc3c-81b7-462c-a902-671f933e4a10\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " Nov 03 17:49:11.994232 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.994162 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\" (UniqueName: \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\") pod \"c333dc3c-81b7-462c-a902-671f933e4a10\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " Nov 03 17:49:11.994337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.994232 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\") pod \"c333dc3c-81b7-462c-a902-671f933e4a10\" (UID: \"c333dc3c-81b7-462c-a902-671f933e4a10\") " Nov 03 17:49:11.994715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.994682 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca" pod "c333dc3c-81b7-462c-a902-671f933e4a10" (UID: "c333dc3c-81b7-462c-a902-671f933e4a10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:11.997650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:11.997609 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls" pod "c333dc3c-81b7-462c-a902-671f933e4a10" (UID: "c333dc3c-81b7-462c-a902-671f933e4a10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:12.000194 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.000164 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs" pod "c333dc3c-81b7-462c-a902-671f933e4a10" (UID: "c333dc3c-81b7-462c-a902-671f933e4a10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:12.006333 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.006305 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z" pod "c333dc3c-81b7-462c-a902-671f933e4a10" (UID: "c333dc3c-81b7-462c-a902-671f933e4a10"). InnerVolumeSpecName "kube-api-access-p8q2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:12.095643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.095569 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:12.095643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.095597 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c333dc3c-81b7-462c-a902-671f933e4a10-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:12.095643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.095608 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8q2z\" (UniqueName: \"kubernetes.io/projected/c333dc3c-81b7-462c-a902-671f933e4a10-kube-api-access-p8q2z\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:12.095643 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.095618 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c333dc3c-81b7-462c-a902-671f933e4a10-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:12.278338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.278307 4885 generic.go:356] "Generic (PLEG): container finished" podID="c333dc3c-81b7-462c-a902-671f933e4a10" containerID="efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041" exitCode=0 Nov 03 17:49:12.278512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.278373 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" Nov 03 17:49:12.278512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.278385 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" event={"ID":"c333dc3c-81b7-462c-a902-671f933e4a10","Type":"ContainerDied","Data":"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041"} Nov 03 17:49:12.278512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.278442 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc" event={"ID":"c333dc3c-81b7-462c-a902-671f933e4a10","Type":"ContainerDied","Data":"a704d5e4c4258f9c9484a4361a220857707fe9c7e551f1d4f4b1d02e864249a2"} Nov 03 17:49:12.278512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.278463 4885 scope.go:122] "RemoveContainer" containerID="efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041" Nov 03 17:49:12.288312 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.288294 4885 scope.go:122] "RemoveContainer" containerID="efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041" Nov 03 17:49:12.288622 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:12.288603 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041\": container with ID starting with efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041 not found: ID does not exist" containerID="efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041" Nov 03 17:49:12.288691 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.288629 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041"} err="failed to get container status \"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041\": rpc error: code = NotFound desc = could not find container \"efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041\": container with ID starting with efe7c1440228a75baebca2d8db26a94de92f5c294744ce79695a74d4dceb3041 not found: ID does not exist" Nov 03 17:49:12.475157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.475121 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:49:12.527378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:12.527338 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b3c7bad-d897b844b-24hdc"] Nov 03 17:49:13.662239 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:13.662199 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c333dc3c-81b7-462c-a902-671f933e4a10" path="/var/lib/kubelet/pods/c333dc3c-81b7-462c-a902-671f933e4a10/volumes" Nov 03 17:49:13.667276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:13.667252 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz"] Nov 03 17:49:13.674324 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:13.674297 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b3c7bad-r7hzz"] Nov 03 17:49:13.740814 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:13.740780 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:49:13.741182 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:13.741142 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" containerID="cri-o://87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262" gracePeriod=30 Nov 03 17:49:15.227469 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227436 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227837 4885 cpu_manager.go:414] "containerMap: removing container" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227847 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227865 4885 cpu_manager.go:414] "containerMap: removing container" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227871 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227881 4885 cpu_manager.go:414] "containerMap: removing container" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227885 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227899 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c333dc3c-81b7-462c-a902-671f933e4a10" containerName="maas-api" Nov 03 17:49:15.227948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227904 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c333dc3c-81b7-462c-a902-671f933e4a10" containerName="maas-api" Nov 03 17:49:15.228364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227968 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b796743-59cf-464d-90a7-b1b0226f9748" containerName="main" Nov 03 17:49:15.228364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227976 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a02bf5-bb41-40a7-92fb-68acb2d5cca8" containerName="main" Nov 03 17:49:15.228364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227983 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e5f376-8ed7-49b4-8779-12218aa50387" containerName="main" Nov 03 17:49:15.228364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.227990 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c333dc3c-81b7-462c-a902-671f933e4a10" containerName="maas-api" Nov 03 17:49:15.243956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.243916 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.288729 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.288696 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:49:15.337827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.337796 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-382697-openshift-default-dockercfg-gww5g\"" Nov 03 17:49:15.345506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345475 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345534 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345638 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345669 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345716 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345696 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345800 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345820 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345837 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp84d\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.345978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.345868 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.446707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.446666 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.447585 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.447560 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.447783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.447766 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.447919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.447906 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.448028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.448014 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.448272 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.448251 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp84d\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.456099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.448535 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.457865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.457842 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.458027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.458012 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.458321 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.457374 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.461334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.456579 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.468137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.461914 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.468298 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.457170 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.468397 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.464530 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.468485 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.464679 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.468885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.468864 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.538965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.538894 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp84d\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.555888 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.555850 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\") pod \"e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.560109 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.560040 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:15.669128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:15.668669 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98c7165-48c0-4dd8-98b0-a8dcf1da5264" path="/var/lib/kubelet/pods/c98c7165-48c0-4dd8-98b0-a8dcf1da5264/volumes" Nov 03 17:49:16.070817 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:49:16.070789 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a9d20c_c72b_4e0d_946a_2b7abd2d02e5.slice/crio-a9496e70479b6e9600926b0af9a238b9065a7b110466406cda3012036e541d5f WatchSource:0}: Error finding container a9496e70479b6e9600926b0af9a238b9065a7b110466406cda3012036e541d5f: Status 404 returned error can't find the container with id a9496e70479b6e9600926b0af9a238b9065a7b110466406cda3012036e541d5f Nov 03 17:49:16.073024 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.073003 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:49:16.073111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.073061 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:49:16.073153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.073136 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:49:16.073183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.073164 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:49:16.308541 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.308503 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerStarted","Data":"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7"} Nov 03 17:49:16.308942 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.308568 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerStarted","Data":"a9496e70479b6e9600926b0af9a238b9065a7b110466406cda3012036e541d5f"} Nov 03 17:49:16.433492 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.433445 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podStartSLOduration=1.433429686 podStartE2EDuration="1.433429686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:16.431193084 +0000 UTC m=+1898.885882108" watchObservedRunningTime="2026-11-03 17:49:16.433429686 +0000 UTC m=+1898.888118710" Nov 03 17:49:16.574042 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:16.574011 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:17.566711 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:17.566683 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:18.318208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:18.318177 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:18.319188 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:18.319166 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:49:18.606564 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:18.606476 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:49:19.020284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.020261 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:49:19.106864 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.106829 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.106895 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.106917 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.106946 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.106978 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107038 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107037 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107167 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.107339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107175 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107252 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.107339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107281 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107323 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\") pod \"127f3405-3e94-4b95-8bd9-2717865220e3\" (UID: \"127f3405-3e94-4b95-8bd9-2717865220e3\") " Nov 03 17:49:19.107564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107409 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.107564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107429 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:19.107564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107550 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.107857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107834 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.107857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107858 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.108023 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107872 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.108023 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107886 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127f3405-3e94-4b95-8bd9-2717865220e3-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.108023 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.107898 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.109468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.109443 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:49:19.109555 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.109487 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.109768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.109751 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:19.109768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.109760 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq" pod "127f3405-3e94-4b95-8bd9-2717865220e3" (UID: "127f3405-3e94-4b95-8bd9-2717865220e3"). InnerVolumeSpecName "kube-api-access-bc4bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:19.209760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.209720 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.209760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.209758 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127f3405-3e94-4b95-8bd9-2717865220e3-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.209949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.209774 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc4bq\" (UniqueName: \"kubernetes.io/projected/127f3405-3e94-4b95-8bd9-2717865220e3-kube-api-access-bc4bq\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.209949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.209789 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127f3405-3e94-4b95-8bd9-2717865220e3-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.325728 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.325695 4885 generic.go:356] "Generic (PLEG): container finished" podID="127f3405-3e94-4b95-8bd9-2717865220e3" containerID="87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262" exitCode=0 Nov 03 17:49:19.325933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.325740 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" Nov 03 17:49:19.325933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.325777 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerDied","Data":"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262"} Nov 03 17:49:19.325933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.325833 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h" event={"ID":"127f3405-3e94-4b95-8bd9-2717865220e3","Type":"ContainerDied","Data":"617e19d65bffc6b7b04780cc80578dbe02664301d54f03a5ed3bf6cf6ad32f9b"} Nov 03 17:49:19.325933 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.325857 4885 scope.go:122] "RemoveContainer" containerID="87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262" Nov 03 17:49:19.341858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.340458 4885 scope.go:122] "RemoveContainer" containerID="0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13" Nov 03 17:49:19.350844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.350813 4885 scope.go:122] "RemoveContainer" containerID="87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262" Nov 03 17:49:19.351218 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:19.351193 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262\": container with ID starting with 87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262 not found: ID does not exist" containerID="87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262" Nov 03 17:49:19.351313 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.351231 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262"} err="failed to get container status \"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262\": rpc error: code = NotFound desc = could not find container \"87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262\": container with ID starting with 87c28b1dde420ff64bf1593da011fa994e7d225ddba10d8aafb1f45a8bc34262 not found: ID does not exist" Nov 03 17:49:19.351313 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.351256 4885 scope.go:122] "RemoveContainer" containerID="0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13" Nov 03 17:49:19.351571 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:19.351551 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13\": container with ID starting with 0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13 not found: ID does not exist" containerID="0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13" Nov 03 17:49:19.351641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.351581 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13"} err="failed to get container status \"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13\": rpc error: code = NotFound desc = could not find container \"0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13\": container with ID starting with 0a509fadcca62fb9116074971eecd47ba244d3722ffe30a59bae4ef8d06b0c13 not found: ID does not exist" Nov 03 17:49:19.500287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.500247 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:49:19.508143 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.508111 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1b3c7bad-openshift-default-6c4f7b67c-hs46h"] Nov 03 17:49:19.661378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:19.661348 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" path="/var/lib/kubelet/pods/127f3405-3e94-4b95-8bd9-2717865220e3/volumes" Nov 03 17:49:29.018432 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:29.018398 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:49:35.357002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:35.356958 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:49:47.992421 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992385 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:49:47.992869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992837 4885 cpu_manager.go:414] "containerMap: removing container" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.992869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992850 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.992869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992858 4885 cpu_manager.go:414] "containerMap: removing container" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.992869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992864 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.992997 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992943 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.992997 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.992952 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="127f3405-3e94-4b95-8bd9-2717865220e3" containerName="istio-proxy" Nov 03 17:49:47.996030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:47.996013 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.045535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.045126 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.045535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.045269 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.045535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.045321 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.045535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.045411 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm6nb\" (UniqueName: \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.066876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.066824 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:49:48.067688 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.067668 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-382697-serving-cert\"" Nov 03 17:49:48.106961 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.106922 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 17:49:48.106961 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.106958 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:49:48.107203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.106964 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:49:48.107203 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.106961 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:49:48.146330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.146297 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm6nb\" (UniqueName: \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.146493 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.146357 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.146577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.146487 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.146577 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:48.146501 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-382697-serving-cert: secret "maas-api-e2e-api-a-382697-serving-cert" not found Nov 03 17:49:48.146577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.146550 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.146727 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:48.146587 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls podName:c0a1f59e-f973-4a03-8987-dd9d552fde71 nodeName:}" failed. No retries permitted until 2026-11-03 17:49:48.646565733 +0000 UTC m=+1931.101254738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls") pod "maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" (UID: "c0a1f59e-f973-4a03-8987-dd9d552fde71") : secret "maas-api-e2e-api-a-382697-serving-cert" not found Nov 03 17:49:48.147147 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.147125 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.149178 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.149152 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.208968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.208937 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm6nb\" (UniqueName: \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.610339 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:49:48.610265 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:49:48.675123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.674672 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.689115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.687610 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") pod \"maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:48.920092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:48.918266 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:49.474352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:49.474299 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:49:49.489600 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:49:49.489357 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a1f59e_f973_4a03_8987_dd9d552fde71.slice/crio-066d61666145afefd33263023728b61e9abcf68aedd6862b197c2e6bc25c881b WatchSource:0}: Error finding container 066d61666145afefd33263023728b61e9abcf68aedd6862b197c2e6bc25c881b: Status 404 returned error can't find the container with id 066d61666145afefd33263023728b61e9abcf68aedd6862b197c2e6bc25c881b Nov 03 17:49:49.588305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:49.588255 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" event={"ID":"c0a1f59e-f973-4a03-8987-dd9d552fde71","Type":"ContainerStarted","Data":"066d61666145afefd33263023728b61e9abcf68aedd6862b197c2e6bc25c881b"} Nov 03 17:49:50.413093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:50.410282 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" containerID="cri-o://bc03386c1141cbe74a6cb7cfe705aef55da6187f4258eb10113098d78751dd0d" gracePeriod=45 Nov 03 17:49:50.660726 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:50.660584 4885 generic.go:356] "Generic (PLEG): container finished" podID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerID="bc03386c1141cbe74a6cb7cfe705aef55da6187f4258eb10113098d78751dd0d" exitCode=0 Nov 03 17:49:50.660726 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:50.660665 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" event={"ID":"fac6a0b8-28ed-458f-ba98-c76124ab5499","Type":"ContainerDied","Data":"bc03386c1141cbe74a6cb7cfe705aef55da6187f4258eb10113098d78751dd0d"} Nov 03 17:49:51.033541 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.033437 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused" Nov 03 17:49:51.564119 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.563207 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628464 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628509 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628583 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628622 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\" (UniqueName: \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628724 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\" (UniqueName: \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.629092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.628796 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\") pod \"fac6a0b8-28ed-458f-ba98-c76124ab5499\" (UID: \"fac6a0b8-28ed-458f-ba98-c76124ab5499\") " Nov 03 17:49:51.630127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.629961 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:51.630361 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.630337 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:51.630848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.630823 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:51.674302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.665755 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "kube-api-access-q5d7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:51.674302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.665890 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:51.678111 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.675467 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm" pod "fac6a0b8-28ed-458f-ba98-c76124ab5499" (UID: "fac6a0b8-28ed-458f-ba98-c76124ab5499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:51.687497 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.687466 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" event={"ID":"c0a1f59e-f973-4a03-8987-dd9d552fde71","Type":"ContainerStarted","Data":"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07"} Nov 03 17:49:51.688167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.688146 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:49:51.690047 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.690028 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" event={"ID":"fac6a0b8-28ed-458f-ba98-c76124ab5499","Type":"ContainerDied","Data":"3db65ab582b0d9d927e9473582743d9b632a3569d71b603a990647e48c34ec68"} Nov 03 17:49:51.690196 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.690176 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk" Nov 03 17:49:51.690287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.690197 4885 scope.go:122] "RemoveContainer" containerID="bc03386c1141cbe74a6cb7cfe705aef55da6187f4258eb10113098d78751dd0d" Nov 03 17:49:51.756145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.733940 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.756393 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.756370 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fac6a0b8-28ed-458f-ba98-c76124ab5499-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.756539 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.756524 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5d7h\" (UniqueName: \"kubernetes.io/projected/fac6a0b8-28ed-458f-ba98-c76124ab5499-kube-api-access-q5d7h\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.756649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.756635 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.756753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.756739 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.756935 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.756922 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fac6a0b8-28ed-458f-ba98-c76124ab5499-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:51.826094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.824810 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" podStartSLOduration=3.812986549 podStartE2EDuration="4.824776604s" totalImagesPullingTime="1.011790055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:51.819840139 +0000 UTC m=+1934.274529163" watchObservedRunningTime="2026-11-03 17:49:51.824776604 +0000 UTC m=+1934.279465630" Nov 03 17:49:51.952153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.952116 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:49:51.977208 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:51.977162 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-147834/sub-test-model-147834-kserve-cc6c97b7f-5rnjk"] Nov 03 17:49:53.661912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:53.661879 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" path="/var/lib/kubelet/pods/fac6a0b8-28ed-458f-ba98-c76124ab5499/volumes" Nov 03 17:49:58.708746 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:49:58.708713 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:50:00.581281 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:00.581251 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:50:10.793428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.793393 4885 generic.go:356] "Generic (PLEG): container finished" podID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerID="887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7" exitCode=0 Nov 03 17:50:10.793840 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.793469 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerDied","Data":"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7"} Nov 03 17:50:10.793879 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.793844 4885 scope.go:122] "RemoveContainer" containerID="887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7" Nov 03 17:50:10.795177 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.795150 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:50:10.795248 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.795220 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:50:10.795286 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:10.795251 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:50:11.800720 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:11.800688 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerStarted","Data":"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2"} Nov 03 17:50:12.561126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:12.561090 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:50:12.561336 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:12.561315 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:50:12.561405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:12.561374 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 17:50:13.560790 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:13.560749 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:50:13.561214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:13.560817 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 17:50:14.561282 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:14.561244 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:50:14.561764 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:14.561301 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 17:50:15.560675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:15.560639 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:50:15.560865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:15.560688 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 17:50:16.188177 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.188150 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:50:16.188567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.188546 4885 cpu_manager.go:414] "containerMap: removing container" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" Nov 03 17:50:16.188675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.188569 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" Nov 03 17:50:16.188675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.188643 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac6a0b8-28ed-458f-ba98-c76124ab5499" containerName="main" Nov 03 17:50:16.190950 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.190931 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.278476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.278427 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.278476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.278472 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.278735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.278496 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.278735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.278634 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.330520 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.330483 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-1f6be0-serving-cert\"" Nov 03 17:50:16.333751 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.333722 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:50:16.379487 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.379456 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.379640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.379610 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.379719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.379642 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.379719 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.379672 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.379852 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:50:16.379836 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-serving-cert: secret "maas-api-e2e-autores-1f6be0-serving-cert" not found Nov 03 17:50:16.379931 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:50:16.379910 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls podName:4978c6e3-e074-4e97-a890-1dc879002d49 nodeName:}" failed. No retries permitted until 2026-11-03 17:50:16.879886371 +0000 UTC m=+1959.334575392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls") pod "maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" (UID: "4978c6e3-e074-4e97-a890-1dc879002d49") : secret "maas-api-e2e-autores-1f6be0-serving-cert" not found Nov 03 17:50:16.380255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.380232 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.382184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.382167 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.480141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.480017 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.561480 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.561447 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:50:16.561661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.561502 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 03 17:50:16.885965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.885932 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:16.888548 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:16.888519 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") pod \"maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:17.103898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.103860 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:17.483901 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:50:17.483848 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4978c6e3_e074_4e97_a890_1dc879002d49.slice/crio-db2c445aae131293e716074244b5411ffd3b9dc675e57b71a1cd41e69a80c485 WatchSource:0}: Error finding container db2c445aae131293e716074244b5411ffd3b9dc675e57b71a1cd41e69a80c485: Status 404 returned error can't find the container with id db2c445aae131293e716074244b5411ffd3b9dc675e57b71a1cd41e69a80c485 Nov 03 17:50:17.487029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.487009 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:50:17.565124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.565095 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:50:17.565441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.565420 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:50:17.566061 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.566042 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:50:17.857054 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:17.856970 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" event={"ID":"4978c6e3-e074-4e97-a890-1dc879002d49","Type":"ContainerStarted","Data":"db2c445aae131293e716074244b5411ffd3b9dc675e57b71a1cd41e69a80c485"} Nov 03 17:50:18.605892 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:50:18.605850 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:50:18.863450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:18.863366 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" event={"ID":"4978c6e3-e074-4e97-a890-1dc879002d49","Type":"ContainerStarted","Data":"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be"} Nov 03 17:50:18.985175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:18.985129 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" podStartSLOduration=2.101585073 podStartE2EDuration="2.985114846s" totalImagesPullingTime="883.529773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:18.983862965 +0000 UTC m=+1961.438551992" watchObservedRunningTime="2026-11-03 17:50:18.985114846 +0000 UTC m=+1961.439803894" Nov 03 17:50:19.868924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:19.868893 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:25.883937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:25.883907 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:50:26.414927 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:50:26.414893 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:50:48.605034 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:50:48.604997 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:51:06.011406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.011322 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:51:06.015359 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.015342 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.071196 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.071167 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:51:06.076757 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.076729 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.076921 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.076900 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.076983 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.076967 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.077116 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.077055 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc989\" (UniqueName: \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.077206 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.077184 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.077247 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.077229 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.119914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.119881 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1f6be0\"/\"openshift-service-ca.crt\"" Nov 03 17:51:06.120114 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.119971 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1f6be0\"/\"default-dockercfg-mpj5m\"" Nov 03 17:51:06.120303 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.120285 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1f6be0\"/\"autores-model-1f6be0-kserve-self-signed-certs\"" Nov 03 17:51:06.120361 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.120319 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1f6be0\"/\"kube-root-ca.crt\"" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.178763 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.178916 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.178956 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc989\" (UniqueName: \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.179013 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.179045 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.179143 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.179793 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.180227 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183173 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.181429 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.183962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.183907 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.216129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.214748 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.245026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.244992 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc989\" (UniqueName: \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\") pod \"autores-model-1f6be0-kserve-7c5cd6858f-x4crk\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.326794 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.326701 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:06.698148 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:51:06.698115 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfca8cb32_e5a0_406f_8b22_fe4281b0381e.slice/crio-1ad119d88610aee002401e47278c0993ea3ee7ef8dffd826762ddc7f01c165dc WatchSource:0}: Error finding container 1ad119d88610aee002401e47278c0993ea3ee7ef8dffd826762ddc7f01c165dc: Status 404 returned error can't find the container with id 1ad119d88610aee002401e47278c0993ea3ee7ef8dffd826762ddc7f01c165dc Nov 03 17:51:06.700035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:06.700013 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:51:07.131922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:07.131893 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" event={"ID":"fca8cb32-e5a0-406f-8b22-fe4281b0381e","Type":"ContainerStarted","Data":"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45"} Nov 03 17:51:07.132310 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:07.131934 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" event={"ID":"fca8cb32-e5a0-406f-8b22-fe4281b0381e","Type":"ContainerStarted","Data":"1ad119d88610aee002401e47278c0993ea3ee7ef8dffd826762ddc7f01c165dc"} Nov 03 17:51:07.253723 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:07.253658 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" podStartSLOduration=2.253642569 podStartE2EDuration="2.253642569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:07.252315549 +0000 UTC m=+2009.707004573" watchObservedRunningTime="2026-11-03 17:51:07.253642569 +0000 UTC m=+2009.708331587" Nov 03 17:51:07.326804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:07.326769 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:12.735715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:12.735679 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:51:16.327481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:16.327438 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:16.344150 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:16.344121 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:17.070578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.070544 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:51:17.074157 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.074134 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.132570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.132531 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e095fe-serving-cert\"" Nov 03 17:51:17.147172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.146803 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:51:17.218952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.218087 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.218952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.218173 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.218952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.218219 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7g4\" (UniqueName: \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.218952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.218257 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.230098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.230032 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:51:17.319876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.319844 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.319892 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.319937 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7g4\" (UniqueName: \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.320083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.319971 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.320263 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:51:17.320092 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-serving-cert: secret "maas-api-e2e-api-b-e095fe-serving-cert" not found Nov 03 17:51:17.320263 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:51:17.320163 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls podName:1b386b98-caad-4149-965c-f0e22b3ed4fc nodeName:}" failed. No retries permitted until 2026-11-03 17:51:17.820141683 +0000 UTC m=+2020.274830699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls") pod "maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" (UID: "1b386b98-caad-4149-965c-f0e22b3ed4fc") : secret "maas-api-e2e-api-b-e095fe-serving-cert" not found Nov 03 17:51:17.320741 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.320681 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.322649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.322620 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.393848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.393821 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7g4\" (UniqueName: \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.837005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.836961 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.839514 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.839495 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") pod \"maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:17.988591 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:17.988537 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:18.410437 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:51:18.410403 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b386b98_caad_4149_965c_f0e22b3ed4fc.slice/crio-2c8ed49bef69f268529feb596c5f58ace7e4cb9bd9aa19bdb77a942610d23ffa WatchSource:0}: Error finding container 2c8ed49bef69f268529feb596c5f58ace7e4cb9bd9aa19bdb77a942610d23ffa: Status 404 returned error can't find the container with id 2c8ed49bef69f268529feb596c5f58ace7e4cb9bd9aa19bdb77a942610d23ffa Nov 03 17:51:18.412854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:18.412771 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:51:18.605796 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:51:18.605759 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:51:19.228524 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:19.228489 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" event={"ID":"1b386b98-caad-4149-965c-f0e22b3ed4fc","Type":"ContainerStarted","Data":"2c8ed49bef69f268529feb596c5f58ace7e4cb9bd9aa19bdb77a942610d23ffa"} Nov 03 17:51:20.236377 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:20.236346 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" event={"ID":"1b386b98-caad-4149-965c-f0e22b3ed4fc","Type":"ContainerStarted","Data":"0c58ace83a64493875a7accc721e51274c9d9b5f76e76e507deae313af56e2b9"} Nov 03 17:51:20.236848 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:20.236470 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:20.394300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:20.394240 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" podStartSLOduration=2.525383024 podStartE2EDuration="3.394225347s" totalImagesPullingTime="868.842323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:20.35682467 +0000 UTC m=+2022.811513694" watchObservedRunningTime="2026-11-03 17:51:20.394225347 +0000 UTC m=+2022.848914370" Nov 03 17:51:26.247356 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:26.247311 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:51:37.969490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:37.969463 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:51:37.973364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:37.973345 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.027740 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.027714 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e095fe-openshift-default-dockercfg-lcjc7\"" Nov 03 17:51:38.065672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065643 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.065851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065691 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.065851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065719 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.065851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065835 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.065971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065872 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.065971 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.065891 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.066059 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.066035 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.066138 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.066116 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.066213 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.066189 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.081297 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.081265 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:51:38.167496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167449 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167519 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167584 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167625 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167704 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167779 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.167835 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167808 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167839 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167882 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168055 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.167900 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168326 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.168258 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168393 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.168370 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168648 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.168574 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.168796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.168777 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.170277 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.170253 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.170379 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.170296 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.231711 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.231630 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.231891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.231714 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.288061 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.288026 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:38.710618 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:51:38.710589 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae2d7f2_46e2_4d35_a63c_ff283fb5a346.slice/crio-f8e1cb231bafa05a630c78aa723f1d0707ee07660671bf2d5df7e92412e8b8ad WatchSource:0}: Error finding container f8e1cb231bafa05a630c78aa723f1d0707ee07660671bf2d5df7e92412e8b8ad: Status 404 returned error can't find the container with id f8e1cb231bafa05a630c78aa723f1d0707ee07660671bf2d5df7e92412e8b8ad Nov 03 17:51:38.712470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.712449 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:51:38.713708 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.713674 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:38.713768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.713752 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:38.713808 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:38.713798 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:39.343350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:39.343310 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" event={"ID":"dae2d7f2-46e2-4d35-a63c-ff283fb5a346","Type":"ContainerStarted","Data":"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c"} Nov 03 17:51:39.343799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:39.343379 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" event={"ID":"dae2d7f2-46e2-4d35-a63c-ff283fb5a346","Type":"ContainerStarted","Data":"f8e1cb231bafa05a630c78aa723f1d0707ee07660671bf2d5df7e92412e8b8ad"} Nov 03 17:51:40.289005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:40.288966 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:40.290517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:40.290493 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:51:40.290641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:40.290541 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 17:51:41.289054 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:41.289017 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:51:41.289453 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:41.289089 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 17:51:42.288714 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:42.288674 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:51:42.288885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:42.288734 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 17:51:42.370564 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:42.370512 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podStartSLOduration=5.370498298 podStartE2EDuration="5.370498298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:39.512669198 +0000 UTC m=+2041.967358226" watchObservedRunningTime="2026-11-03 17:51:42.370498298 +0000 UTC m=+2044.825187324" Nov 03 17:51:42.370960 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:42.370702 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:51:43.288695 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:43.288659 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:51:43.288911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:43.288712 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 17:51:44.288664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:44.288630 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:51:44.289045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:44.288693 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Nov 03 17:51:45.292949 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:45.292915 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:45.293398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:45.293211 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:45.294022 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:45.294005 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:51:47.485404 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:51:47.485370 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:51:48.606149 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:51:48.606114 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:52:02.504994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.504930 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" podUID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" containerName="main" containerID="cri-o://e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45" gracePeriod=45 Nov 03 17:52:02.836102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.836051 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:52:02.995540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995505 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.995766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995661 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.995766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995727 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\" (UniqueName: \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.995909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995815 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\" (UniqueName: \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.995909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995884 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.996029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995945 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\") pod \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\" (UID: \"fca8cb32-e5a0-406f-8b22-fe4281b0381e\") " Nov 03 17:52:02.996029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.995986 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:02.996261 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.996232 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:02.996441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.996423 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:02.996517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.996448 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:02.996600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.996524 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:02.999255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.999226 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:02.999350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.999293 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "kube-api-access-hc989". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:02.999350 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:02.999313 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs" pod "fca8cb32-e5a0-406f-8b22-fe4281b0381e" (UID: "fca8cb32-e5a0-406f-8b22-fe4281b0381e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:03.098144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.098050 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:03.098144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.098091 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fca8cb32-e5a0-406f-8b22-fe4281b0381e-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:03.098144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.098100 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fca8cb32-e5a0-406f-8b22-fe4281b0381e-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:03.098144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.098110 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc989\" (UniqueName: \"kubernetes.io/projected/fca8cb32-e5a0-406f-8b22-fe4281b0381e-kube-api-access-hc989\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:03.484747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.484713 4885 generic.go:356] "Generic (PLEG): container finished" podID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" containerID="e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45" exitCode=0 Nov 03 17:52:03.484939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.484754 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" event={"ID":"fca8cb32-e5a0-406f-8b22-fe4281b0381e","Type":"ContainerDied","Data":"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45"} Nov 03 17:52:03.484939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.484773 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" Nov 03 17:52:03.484939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.484790 4885 scope.go:122] "RemoveContainer" containerID="e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45" Nov 03 17:52:03.484939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.484781 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk" event={"ID":"fca8cb32-e5a0-406f-8b22-fe4281b0381e","Type":"ContainerDied","Data":"1ad119d88610aee002401e47278c0993ea3ee7ef8dffd826762ddc7f01c165dc"} Nov 03 17:52:03.495820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.495799 4885 scope.go:122] "RemoveContainer" containerID="e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45" Nov 03 17:52:03.496106 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:03.496081 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45\": container with ID starting with e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45 not found: ID does not exist" containerID="e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45" Nov 03 17:52:03.496162 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.496118 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45"} err="failed to get container status \"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45\": rpc error: code = NotFound desc = could not find container \"e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45\": container with ID starting with e311ee41b4bf185f03c34464bed9a486ae97701593274eb190e916694ca74a45 not found: ID does not exist" Nov 03 17:52:03.663000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.662964 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:52:03.713495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:03.713466 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1f6be0/autores-model-1f6be0-kserve-7c5cd6858f-x4crk"] Nov 03 17:52:05.662339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:05.662296 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" path="/var/lib/kubelet/pods/fca8cb32-e5a0-406f-8b22-fe4281b0381e/volumes" Nov 03 17:52:06.066406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:06.066314 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:52:17.542280 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.542241 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:52:17.542775 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.542524 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" podUID="4978c6e3-e074-4e97-a890-1dc879002d49" containerName="maas-api" containerID="cri-o://2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be" gracePeriod=30 Nov 03 17:52:17.892619 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.892583 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:52:17.993102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.986603 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\") pod \"4978c6e3-e074-4e97-a890-1dc879002d49\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " Nov 03 17:52:17.993102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.986656 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\") pod \"4978c6e3-e074-4e97-a890-1dc879002d49\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " Nov 03 17:52:17.993102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.986717 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") pod \"4978c6e3-e074-4e97-a890-1dc879002d49\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " Nov 03 17:52:17.993102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:17.986783 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\") pod \"4978c6e3-e074-4e97-a890-1dc879002d49\" (UID: \"4978c6e3-e074-4e97-a890-1dc879002d49\") " Nov 03 17:52:18.003669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.000146 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs" pod "4978c6e3-e074-4e97-a890-1dc879002d49" (UID: "4978c6e3-e074-4e97-a890-1dc879002d49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:18.009506 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.009471 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca" pod "4978c6e3-e074-4e97-a890-1dc879002d49" (UID: "4978c6e3-e074-4e97-a890-1dc879002d49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:18.027307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.019373 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls" pod "4978c6e3-e074-4e97-a890-1dc879002d49" (UID: "4978c6e3-e074-4e97-a890-1dc879002d49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:18.043104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.042289 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j" pod "4978c6e3-e074-4e97-a890-1dc879002d49" (UID: "4978c6e3-e074-4e97-a890-1dc879002d49"). InnerVolumeSpecName "kube-api-access-ssz8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:18.088667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.088625 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:18.088667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.088665 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4978c6e3-e074-4e97-a890-1dc879002d49-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:18.088912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.088681 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssz8j\" (UniqueName: \"kubernetes.io/projected/4978c6e3-e074-4e97-a890-1dc879002d49-kube-api-access-ssz8j\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:18.088912 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.088697 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4978c6e3-e074-4e97-a890-1dc879002d49-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:18.595681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.595643 4885 generic.go:356] "Generic (PLEG): container finished" podID="4978c6e3-e074-4e97-a890-1dc879002d49" containerID="2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be" exitCode=0 Nov 03 17:52:18.596171 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.595697 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" event={"ID":"4978c6e3-e074-4e97-a890-1dc879002d49","Type":"ContainerDied","Data":"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be"} Nov 03 17:52:18.596329 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.595779 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" Nov 03 17:52:18.596464 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.596443 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89" event={"ID":"4978c6e3-e074-4e97-a890-1dc879002d49","Type":"ContainerDied","Data":"db2c445aae131293e716074244b5411ffd3b9dc675e57b71a1cd41e69a80c485"} Nov 03 17:52:18.596611 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.596552 4885 scope.go:122] "RemoveContainer" containerID="2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be" Nov 03 17:52:18.624035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.624015 4885 scope.go:122] "RemoveContainer" containerID="2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be" Nov 03 17:52:18.624385 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:18.624366 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be\": container with ID starting with 2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be not found: ID does not exist" containerID="2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be" Nov 03 17:52:18.624461 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.624401 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be"} err="failed to get container status \"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be\": rpc error: code = NotFound desc = could not find container \"2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be\": container with ID starting with 2206675597db4cbcf5d580272de50eea76d0253da32a0abc3fc5637d8cf796be not found: ID does not exist" Nov 03 17:52:18.638835 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:18.638805 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:52:18.774237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.774197 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:52:18.783820 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:18.783792 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-1f6be0-56bb79f4f-5qq89"] Nov 03 17:52:19.662003 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:19.661968 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4978c6e3-e074-4e97-a890-1dc879002d49" path="/var/lib/kubelet/pods/4978c6e3-e074-4e97-a890-1dc879002d49/volumes" Nov 03 17:52:23.354753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.354719 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:52:23.355207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355193 4885 cpu_manager.go:414] "containerMap: removing container" podUID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" containerName="main" Nov 03 17:52:23.355207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355207 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" containerName="main" Nov 03 17:52:23.355308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355235 4885 cpu_manager.go:414] "containerMap: removing container" podUID="4978c6e3-e074-4e97-a890-1dc879002d49" containerName="maas-api" Nov 03 17:52:23.355308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355241 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4978c6e3-e074-4e97-a890-1dc879002d49" containerName="maas-api" Nov 03 17:52:23.355395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355320 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="4978c6e3-e074-4e97-a890-1dc879002d49" containerName="maas-api" Nov 03 17:52:23.355395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.355337 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca8cb32-e5a0-406f-8b22-fe4281b0381e" containerName="main" Nov 03 17:52:23.359887 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.359860 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.455459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.455428 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:52:23.470567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470538 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470592 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470680 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470705 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470721 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470870 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470909 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.470965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.470932 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.471155 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.471059 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdtd\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572086 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572029 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572136 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572166 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572188 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572273 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572323 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572365 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572498 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572547 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572510 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdtd\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572660 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572549 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572758 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.572891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572854 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.573045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572979 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.573045 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.572993 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.574676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.574657 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.574927 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.574907 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.635473 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.635444 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.635712 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.635685 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdtd\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\") pod \"e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:23.672586 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:23.672553 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:24.118532 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.112967 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:52:24.121630 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.121566 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:24.121748 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.121654 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:24.121748 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.121693 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:24.162001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.161971 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:52:24.162297 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.162270 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" podUID="c0a1f59e-f973-4a03-8987-dd9d552fde71" containerName="maas-api" containerID="cri-o://aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07" gracePeriod=30 Nov 03 17:52:24.485015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.484991 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:52:24.601599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.601394 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\") pod \"c0a1f59e-f973-4a03-8987-dd9d552fde71\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " Nov 03 17:52:24.601599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.601483 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\") pod \"c0a1f59e-f973-4a03-8987-dd9d552fde71\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " Nov 03 17:52:24.601599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.601522 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") pod \"c0a1f59e-f973-4a03-8987-dd9d552fde71\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " Nov 03 17:52:24.601890 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.601784 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca" pod "c0a1f59e-f973-4a03-8987-dd9d552fde71" (UID: "c0a1f59e-f973-4a03-8987-dd9d552fde71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:24.602202 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.602173 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\" (UniqueName: \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\") pod \"c0a1f59e-f973-4a03-8987-dd9d552fde71\" (UID: \"c0a1f59e-f973-4a03-8987-dd9d552fde71\") " Nov 03 17:52:24.602636 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.602602 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a1f59e-f973-4a03-8987-dd9d552fde71-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:24.604896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.604874 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls" pod "c0a1f59e-f973-4a03-8987-dd9d552fde71" (UID: "c0a1f59e-f973-4a03-8987-dd9d552fde71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:24.605030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.604970 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb" pod "c0a1f59e-f973-4a03-8987-dd9d552fde71" (UID: "c0a1f59e-f973-4a03-8987-dd9d552fde71"). InnerVolumeSpecName "kube-api-access-vm6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:24.609956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.606021 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs" pod "c0a1f59e-f973-4a03-8987-dd9d552fde71" (UID: "c0a1f59e-f973-4a03-8987-dd9d552fde71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:24.657899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.657854 4885 generic.go:356] "Generic (PLEG): container finished" podID="c0a1f59e-f973-4a03-8987-dd9d552fde71" containerID="aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07" exitCode=0 Nov 03 17:52:24.658121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.657971 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" event={"ID":"c0a1f59e-f973-4a03-8987-dd9d552fde71","Type":"ContainerDied","Data":"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07"} Nov 03 17:52:24.658121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.658013 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" event={"ID":"c0a1f59e-f973-4a03-8987-dd9d552fde71","Type":"ContainerDied","Data":"066d61666145afefd33263023728b61e9abcf68aedd6862b197c2e6bc25c881b"} Nov 03 17:52:24.658121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.658041 4885 scope.go:122] "RemoveContainer" containerID="aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07" Nov 03 17:52:24.658302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.658141 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz" Nov 03 17:52:24.665645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.665615 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" event={"ID":"801aa294-d130-4e9d-9188-09017008adb1","Type":"ContainerStarted","Data":"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554"} Nov 03 17:52:24.665762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.665669 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" event={"ID":"801aa294-d130-4e9d-9188-09017008adb1","Type":"ContainerStarted","Data":"31c94cab08e9d9346d673695669707d3918f9b8abc9c2f8ee3e6661e8d683153"} Nov 03 17:52:24.676898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.676870 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:24.685911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.681560 4885 scope.go:122] "RemoveContainer" containerID="aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07" Nov 03 17:52:24.685911 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:24.681997 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07\": container with ID starting with aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07 not found: ID does not exist" containerID="aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07" Nov 03 17:52:24.685911 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.682030 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07"} err="failed to get container status \"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07\": rpc error: code = NotFound desc = could not find container \"aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07\": container with ID starting with aa1eeafdc7cf0146df3cfae3c944a0a6fcacd0fb0952c9dc0517c6ce5a51ad07 not found: ID does not exist" Nov 03 17:52:24.704423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.704381 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm6nb\" (UniqueName: \"kubernetes.io/projected/c0a1f59e-f973-4a03-8987-dd9d552fde71-kube-api-access-vm6nb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:24.704423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.704418 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:24.704656 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.704433 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0a1f59e-f973-4a03-8987-dd9d552fde71-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:24.852172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:24.841904 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podStartSLOduration=1.841883859 podStartE2EDuration="1.841883859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:24.84015664 +0000 UTC m=+2087.294845666" watchObservedRunningTime="2026-11-03 17:52:24.841883859 +0000 UTC m=+2087.296572883" Nov 03 17:52:25.005991 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:25.005947 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:52:25.011443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:25.011405 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-382697-58c44fb8ff-ljbcz"] Nov 03 17:52:25.661681 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:25.661637 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a1f59e-f973-4a03-8987-dd9d552fde71" path="/var/lib/kubelet/pods/c0a1f59e-f973-4a03-8987-dd9d552fde71/volumes" Nov 03 17:52:25.674640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:25.674614 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:25.674800 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:25.674665 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:26.326092 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:26.325195 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:52:26.673287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:26.673257 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:26.673672 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:26.673309 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:27.673144 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:27.673109 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:27.673306 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:27.673168 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:28.673318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:28.673282 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:28.673688 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:28.673338 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:29.258786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:29.258752 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:52:29.259323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:29.259264 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" containerID="cri-o://c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2" gracePeriod=30 Nov 03 17:52:29.673518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:29.673490 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:29.673884 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:29.673536 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:30.673043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:30.673014 4885 patch_prober.go:29] interesting pod/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:52:30.673214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:30.673097 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 17:52:30.813600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:30.813560 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:52:30.814098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:30.813861 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" containerID="cri-o://215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf" gracePeriod=30 Nov 03 17:52:31.677682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:31.677656 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:31.677935 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:31.677910 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:31.678608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:31.678594 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:52:32.567917 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:32.567873 4885 patch_prober.go:29] interesting pod/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:52:32.568413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:32.567928 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:52:34.284479 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.284433 4885 patch_prober.go:29] interesting pod/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:52:34.284887 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.284495 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:52:34.524880 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.524852 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:52:34.592364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592285 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592384 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592413 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592458 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592522 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592579 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592643 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592671 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592753 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:34.592900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592659 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:34.592900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592799 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\") pod \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\" (UID: \"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5\") " Nov 03 17:52:34.592900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592802 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:34.593131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592811 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:34.593131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.592981 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:34.593512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.593480 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.593512 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.593506 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.593704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.593521 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.593704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.593534 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.593704 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.593548 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.594865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.594834 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:52:34.594865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.594857 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:34.595007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.594907 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:34.595080 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.595023 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d" pod "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" (UID: "a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5"). InnerVolumeSpecName "kube-api-access-tp84d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:34.694804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.694774 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp84d\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-kube-api-access-tp84d\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.694804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.694802 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.694804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.694812 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.695035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.694821 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:34.718649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.718613 4885 generic.go:356] "Generic (PLEG): container finished" podID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerID="c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2" exitCode=0 Nov 03 17:52:34.718821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.718708 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" Nov 03 17:52:34.718821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.718720 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerDied","Data":"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2"} Nov 03 17:52:34.718821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.718792 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs" event={"ID":"a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5","Type":"ContainerDied","Data":"a9496e70479b6e9600926b0af9a238b9065a7b110466406cda3012036e541d5f"} Nov 03 17:52:34.718821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.718816 4885 scope.go:122] "RemoveContainer" containerID="c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2" Nov 03 17:52:34.728579 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.728466 4885 scope.go:122] "RemoveContainer" containerID="887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7" Nov 03 17:52:34.739415 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.739395 4885 scope.go:122] "RemoveContainer" containerID="c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2" Nov 03 17:52:34.739715 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:34.739696 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2\": container with ID starting with c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2 not found: ID does not exist" containerID="c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2" Nov 03 17:52:34.739813 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.739722 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2"} err="failed to get container status \"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2\": rpc error: code = NotFound desc = could not find container \"c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2\": container with ID starting with c1f99f87ff27f124d189c9576138f7aec36a88fc06b8a7476e63ed0d500679f2 not found: ID does not exist" Nov 03 17:52:34.739813 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.739744 4885 scope.go:122] "RemoveContainer" containerID="887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7" Nov 03 17:52:34.740027 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:34.740003 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7\": container with ID starting with 887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7 not found: ID does not exist" containerID="887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7" Nov 03 17:52:34.740102 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.740036 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7"} err="failed to get container status \"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7\": rpc error: code = NotFound desc = could not find container \"887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7\": container with ID starting with 887f6dd904e4cd72014ca4e5f6c6e1e65c8f59d889fbeb62fbab8dd0ffbc12c7 not found: ID does not exist" Nov 03 17:52:34.889624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.889589 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:52:34.897088 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:34.897036 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-382697-openshift-default-8677b9c4ff-cmhgs"] Nov 03 17:52:35.661511 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:35.661474 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" path="/var/lib/kubelet/pods/a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5/volumes" Nov 03 17:52:36.074020 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.073985 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:52:36.115475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115439 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115485 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115529 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115703 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115771 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115797 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115846 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.115919 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115872 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:36.116033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115939 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.116033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.115984 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\") pod \"768929a4-275d-49b7-9356-17aedbbfb25c\" (UID: \"768929a4-275d-49b7-9356-17aedbbfb25c\") " Nov 03 17:52:36.116258 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116232 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:36.116431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116414 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:36.116508 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116492 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.116570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116517 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.116570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116552 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.116671 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116576 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:36.117246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.116847 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:36.117985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.117963 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:36.118321 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.118301 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "kube-api-access-bpbhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:36.118515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.118496 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:36.118830 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.118813 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo" pod "768929a4-275d-49b7-9356-17aedbbfb25c" (UID: "768929a4-275d-49b7-9356-17aedbbfb25c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217498 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217526 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbhp\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-kube-api-access-bpbhp\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217537 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/768929a4-275d-49b7-9356-17aedbbfb25c-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217545 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/768929a4-275d-49b7-9356-17aedbbfb25c-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217553 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/768929a4-275d-49b7-9356-17aedbbfb25c-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.217575 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.217561 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/768929a4-275d-49b7-9356-17aedbbfb25c-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:36.741489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.741452 4885 generic.go:356] "Generic (PLEG): container finished" podID="768929a4-275d-49b7-9356-17aedbbfb25c" containerID="215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf" exitCode=0 Nov 03 17:52:36.741866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.741503 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" Nov 03 17:52:36.741866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.741502 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" event={"ID":"768929a4-275d-49b7-9356-17aedbbfb25c","Type":"ContainerDied","Data":"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf"} Nov 03 17:52:36.741866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.741613 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2" event={"ID":"768929a4-275d-49b7-9356-17aedbbfb25c","Type":"ContainerDied","Data":"f2d5d13eb023208636114ce16a3c30f9b7298529c6efae9e99285c6c533dcc13"} Nov 03 17:52:36.741866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.741640 4885 scope.go:122] "RemoveContainer" containerID="215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf" Nov 03 17:52:36.751226 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.751197 4885 scope.go:122] "RemoveContainer" containerID="215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf" Nov 03 17:52:36.751493 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:36.751475 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf\": container with ID starting with 215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf not found: ID does not exist" containerID="215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf" Nov 03 17:52:36.751554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.751502 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf"} err="failed to get container status \"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf\": rpc error: code = NotFound desc = could not find container \"215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf\": container with ID starting with 215dfc7b5eca28c3fe5a8d533f9fb324ba108251573e85190796ce59c7ca5edf not found: ID does not exist" Nov 03 17:52:36.912986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.912945 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:52:36.955039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:36.955007 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-llsx2"] Nov 03 17:52:37.662701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:37.662657 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" path="/var/lib/kubelet/pods/768929a4-275d-49b7-9356-17aedbbfb25c/volumes" Nov 03 17:52:40.792862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:40.792830 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:52:48.605705 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:48.605664 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:52:49.055827 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:49.055787 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:52:49.056033 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:49.055990 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" containerID="cri-o://2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957" gracePeriod=30 Nov 03 17:52:54.309455 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.309431 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:52:54.419891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.419864 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.419907 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.419957 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420025 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420052 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420124 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420152 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420196 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420368 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:54.420496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420386 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:54.420496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420427 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\") pod \"c3daef39-fd8a-403b-b772-16266a592579\" (UID: \"c3daef39-fd8a-403b-b772-16266a592579\") " Nov 03 17:52:54.420496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420442 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:54.420699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420677 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:54.421036 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.420981 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:54.421263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.421245 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.421308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.421272 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.421308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.421287 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.421308 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.421300 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3daef39-fd8a-403b-b772-16266a592579-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.421392 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.421314 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.422430 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.422389 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:52:54.422603 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.422580 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "kube-api-access-5wmr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:54.422948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.422930 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:52:54.422948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.422936 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token" pod "c3daef39-fd8a-403b-b772-16266a592579" (UID: "c3daef39-fd8a-403b-b772-16266a592579"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:54.522696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.522664 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3daef39-fd8a-403b-b772-16266a592579-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.522853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.522692 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wmr9\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-kube-api-access-5wmr9\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.522853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.522717 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3daef39-fd8a-403b-b772-16266a592579-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.522853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.522725 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3daef39-fd8a-403b-b772-16266a592579-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:54.832836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.832754 4885 generic.go:356] "Generic (PLEG): container finished" podID="c3daef39-fd8a-403b-b772-16266a592579" containerID="2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957" exitCode=0 Nov 03 17:52:54.832836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.832803 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" Nov 03 17:52:54.832836 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.832814 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" event={"ID":"c3daef39-fd8a-403b-b772-16266a592579","Type":"ContainerDied","Data":"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957"} Nov 03 17:52:54.833121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.832857 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw" event={"ID":"c3daef39-fd8a-403b-b772-16266a592579","Type":"ContainerDied","Data":"3a22750dc27c10c97fc0744ab5fba9ec33db0adece66b380f3ff19ee8d5c0593"} Nov 03 17:52:54.833121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.832876 4885 scope.go:122] "RemoveContainer" containerID="2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957" Nov 03 17:52:54.842184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.842168 4885 scope.go:122] "RemoveContainer" containerID="2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957" Nov 03 17:52:54.842429 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:54.842412 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957\": container with ID starting with 2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957 not found: ID does not exist" containerID="2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957" Nov 03 17:52:54.842473 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:54.842438 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957"} err="failed to get container status \"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957\": rpc error: code = NotFound desc = could not find container \"2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957\": container with ID starting with 2a895143faff967cac310bdece2f991c56bdc92ccee28a4252e218e3fb10b957 not found: ID does not exist" Nov 03 17:52:55.008832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:55.008792 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:52:55.012540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:55.012509 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2355f4-openshift-default-578c7f7b49-25rpw"] Nov 03 17:52:55.662048 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:55.662017 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3daef39-fd8a-403b-b772-16266a592579" path="/var/lib/kubelet/pods/c3daef39-fd8a-403b-b772-16266a592579/volumes" Nov 03 17:52:58.948413 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948381 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:52:58.948800 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948787 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:52:58.948800 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948798 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948812 4885 cpu_manager.go:414] "containerMap: removing container" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948817 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948823 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948828 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948848 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948852 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948862 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c0a1f59e-f973-4a03-8987-dd9d552fde71" containerName="maas-api" Nov 03 17:52:58.948866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948866 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a1f59e-f973-4a03-8987-dd9d552fde71" containerName="maas-api" Nov 03 17:52:58.949129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948920 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:52:58.949129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948927 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="768929a4-275d-49b7-9356-17aedbbfb25c" containerName="istio-proxy" Nov 03 17:52:58.949129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948936 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3daef39-fd8a-403b-b772-16266a592579" containerName="istio-proxy" Nov 03 17:52:58.949129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.948942 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a1f59e-f973-4a03-8987-dd9d552fde71" containerName="maas-api" Nov 03 17:52:58.953499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:58.953480 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.062645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.062613 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-fe8fde-serving-cert\"" Nov 03 17:52:59.064699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.064676 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:52:59.069211 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.069186 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.069330 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.069224 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.069412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.069347 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.069499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.069480 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzvb\" (UniqueName: \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.170984 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.170948 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzvb\" (UniqueName: \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.171172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.171016 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.171172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.171040 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.171172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.171097 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.171325 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:59.171288 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-serving-cert: secret "maas-api-e2e-explicit-fe8fde-serving-cert" not found Nov 03 17:52:59.171383 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:52:59.171371 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls podName:bce56faf-4160-4880-bb6e-6776afa8370d nodeName:}" failed. No retries permitted until 2026-11-03 17:52:59.671351904 +0000 UTC m=+2122.126040923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls") pod "maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" (UID: "bce56faf-4160-4880-bb6e-6776afa8370d") : secret "maas-api-e2e-explicit-fe8fde-serving-cert" not found Nov 03 17:52:59.171772 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.171718 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.173617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.173597 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.238318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.238236 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzvb\" (UniqueName: \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.680606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.680574 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.683245 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.683218 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") pod \"maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:52:59.867315 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:52:59.867281 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:53:00.234137 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:53:00.234091 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce56faf_4160_4880_bb6e_6776afa8370d.slice/crio-35f4fbe5186af350fa4ac0c9007592ea03d61d77067f8b09b5e881218be27a05 WatchSource:0}: Error finding container 35f4fbe5186af350fa4ac0c9007592ea03d61d77067f8b09b5e881218be27a05: Status 404 returned error can't find the container with id 35f4fbe5186af350fa4ac0c9007592ea03d61d77067f8b09b5e881218be27a05 Nov 03 17:53:00.235829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:00.235807 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:53:00.236550 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:00.236526 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:53:00.864663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:00.864623 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" event={"ID":"bce56faf-4160-4880-bb6e-6776afa8370d","Type":"ContainerStarted","Data":"35f4fbe5186af350fa4ac0c9007592ea03d61d77067f8b09b5e881218be27a05"} Nov 03 17:53:01.871516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:01.871477 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" event={"ID":"bce56faf-4160-4880-bb6e-6776afa8370d","Type":"ContainerStarted","Data":"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795"} Nov 03 17:53:01.871920 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:01.871639 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:53:01.994373 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:01.994327 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" podStartSLOduration=3.04959945 podStartE2EDuration="3.994312522s" totalImagesPullingTime="944.713072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:01.992396411 +0000 UTC m=+2124.447085435" watchObservedRunningTime="2026-11-03 17:53:01.994312522 +0000 UTC m=+2124.449001546" Nov 03 17:53:07.619782 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.619750 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:53:07.620260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.619933 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" podUID="1b386b98-caad-4149-965c-f0e22b3ed4fc" containerName="maas-api" containerID="cri-o://0c58ace83a64493875a7accc721e51274c9d9b5f76e76e507deae313af56e2b9" gracePeriod=30 Nov 03 17:53:07.931641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.931177 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:53:07.934168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.933166 4885 generic.go:356] "Generic (PLEG): container finished" podID="1b386b98-caad-4149-965c-f0e22b3ed4fc" containerID="0c58ace83a64493875a7accc721e51274c9d9b5f76e76e507deae313af56e2b9" exitCode=0 Nov 03 17:53:07.934168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.933288 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" event={"ID":"1b386b98-caad-4149-965c-f0e22b3ed4fc","Type":"ContainerDied","Data":"0c58ace83a64493875a7accc721e51274c9d9b5f76e76e507deae313af56e2b9"} Nov 03 17:53:07.958140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.958108 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:53:07.982097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.980169 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\") pod \"1b386b98-caad-4149-965c-f0e22b3ed4fc\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " Nov 03 17:53:07.982097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.980426 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") pod \"1b386b98-caad-4149-965c-f0e22b3ed4fc\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " Nov 03 17:53:07.982097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.980490 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\") pod \"1b386b98-caad-4149-965c-f0e22b3ed4fc\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " Nov 03 17:53:07.982097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:07.980534 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\" (UniqueName: \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\") pod \"1b386b98-caad-4149-965c-f0e22b3ed4fc\" (UID: \"1b386b98-caad-4149-965c-f0e22b3ed4fc\") " Nov 03 17:53:08.031924 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.003272 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca" pod "1b386b98-caad-4149-965c-f0e22b3ed4fc" (UID: "1b386b98-caad-4149-965c-f0e22b3ed4fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:08.033616 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.033516 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs" pod "1b386b98-caad-4149-965c-f0e22b3ed4fc" (UID: "1b386b98-caad-4149-965c-f0e22b3ed4fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:08.037256 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.037210 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4" pod "1b386b98-caad-4149-965c-f0e22b3ed4fc" (UID: "1b386b98-caad-4149-965c-f0e22b3ed4fc"). InnerVolumeSpecName "kube-api-access-gq7g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:08.038009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.037963 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls" pod "1b386b98-caad-4149-965c-f0e22b3ed4fc" (UID: "1b386b98-caad-4149-965c-f0e22b3ed4fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:08.082962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.082866 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b386b98-caad-4149-965c-f0e22b3ed4fc-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:08.082962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.082904 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:08.082962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.082919 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b386b98-caad-4149-965c-f0e22b3ed4fc-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:08.082962 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.082933 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7g4\" (UniqueName: \"kubernetes.io/projected/1b386b98-caad-4149-965c-f0e22b3ed4fc-kube-api-access-gq7g4\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:08.939285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.939248 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" Nov 03 17:53:08.939285 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.939268 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79" event={"ID":"1b386b98-caad-4149-965c-f0e22b3ed4fc","Type":"ContainerDied","Data":"2c8ed49bef69f268529feb596c5f58ace7e4cb9bd9aa19bdb77a942610d23ffa"} Nov 03 17:53:08.939802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:08.939344 4885 scope.go:122] "RemoveContainer" containerID="0c58ace83a64493875a7accc721e51274c9d9b5f76e76e507deae313af56e2b9" Nov 03 17:53:09.109131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:09.109098 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:53:09.161697 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:09.161656 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e095fe-79f59b6b7c-kqd79"] Nov 03 17:53:09.550811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:09.550776 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:53:09.680094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:09.679570 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b386b98-caad-4149-965c-f0e22b3ed4fc" path="/var/lib/kubelet/pods/1b386b98-caad-4149-965c-f0e22b3ed4fc/volumes" Nov 03 17:53:18.036162 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:18.036119 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:53:18.036644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:18.036359 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" containerID="cri-o://3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554" gracePeriod=30 Nov 03 17:53:18.055773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:18.055742 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:53:18.055972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:18.055937 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" containerID="cri-o://c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c" gracePeriod=30 Nov 03 17:53:18.608025 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:18.607985 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:53:22.115218 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:22.115178 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:53:22.169427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:22.169393 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:53:22.664621 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:22.664586 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:53:23.329473 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.329447 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:53:23.356701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.356676 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:53:23.415227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415199 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415227 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415232 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415315 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415351 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415385 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415459 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415508 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415556 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415604 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\") pod \"801aa294-d130-4e9d-9188-09017008adb1\" (UID: \"801aa294-d130-4e9d-9188-09017008adb1\") " Nov 03 17:53:23.415683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415617 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.416007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415709 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.416007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415726 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.416007 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.415780 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:23.416260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.416198 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.416260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.416218 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801aa294-d130-4e9d-9188-09017008adb1-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.416260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.416232 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.416260 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.416246 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.416526 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.416349 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.417866 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.417842 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:23.418220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.418195 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "kube-api-access-drdtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:23.418345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.418295 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.418394 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.418349 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo" pod "801aa294-d130-4e9d-9188-09017008adb1" (UID: "801aa294-d130-4e9d-9188-09017008adb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:53:23.516588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516543 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516610 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516643 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516677 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516706 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516736 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516783 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.516842 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516836 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.517180 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.516866 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\") pod \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\" (UID: \"dae2d7f2-46e2-4d35-a63c-ff283fb5a346\") " Nov 03 17:53:23.517180 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517008 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.517180 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517030 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:23.517323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517179 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.517323 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517190 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.517428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517356 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517378 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517393 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517409 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517428 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517422 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801aa294-d130-4e9d-9188-09017008adb1-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517436 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517453 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517466 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801aa294-d130-4e9d-9188-09017008adb1-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517478 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdtd\" (UniqueName: \"kubernetes.io/projected/801aa294-d130-4e9d-9188-09017008adb1-kube-api-access-drdtd\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.517837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.517511 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.519216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.519196 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "kube-api-access-8qvgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:23.519216 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.519205 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:23.519462 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.519445 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:53:23.519517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.519467 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token" pod "dae2d7f2-46e2-4d35-a63c-ff283fb5a346" (UID: "dae2d7f2-46e2-4d35-a63c-ff283fb5a346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:23.618766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.618738 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.618766 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.618765 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.618952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.618774 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.618952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.618783 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvgn\" (UniqueName: \"kubernetes.io/projected/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-kube-api-access-8qvgn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:23.618952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:23.618792 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae2d7f2-46e2-4d35-a63c-ff283fb5a346-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:24.037571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.037494 4885 generic.go:356] "Generic (PLEG): container finished" podID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerID="c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c" exitCode=0 Nov 03 17:53:24.037571 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.037558 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" Nov 03 17:53:24.037793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.037581 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" event={"ID":"dae2d7f2-46e2-4d35-a63c-ff283fb5a346","Type":"ContainerDied","Data":"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c"} Nov 03 17:53:24.037793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.037635 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7" event={"ID":"dae2d7f2-46e2-4d35-a63c-ff283fb5a346","Type":"ContainerDied","Data":"f8e1cb231bafa05a630c78aa723f1d0707ee07660671bf2d5df7e92412e8b8ad"} Nov 03 17:53:24.037793 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.037664 4885 scope.go:122] "RemoveContainer" containerID="c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c" Nov 03 17:53:24.039134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.039108 4885 generic.go:356] "Generic (PLEG): container finished" podID="801aa294-d130-4e9d-9188-09017008adb1" containerID="3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554" exitCode=0 Nov 03 17:53:24.039242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.039166 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" Nov 03 17:53:24.039242 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.039195 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" event={"ID":"801aa294-d130-4e9d-9188-09017008adb1","Type":"ContainerDied","Data":"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554"} Nov 03 17:53:24.039328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.039250 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw" event={"ID":"801aa294-d130-4e9d-9188-09017008adb1","Type":"ContainerDied","Data":"31c94cab08e9d9346d673695669707d3918f9b8abc9c2f8ee3e6661e8d683153"} Nov 03 17:53:24.046838 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.046823 4885 scope.go:122] "RemoveContainer" containerID="c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c" Nov 03 17:53:24.047145 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:24.047122 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c\": container with ID starting with c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c not found: ID does not exist" containerID="c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c" Nov 03 17:53:24.047218 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.047154 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c"} err="failed to get container status \"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c\": rpc error: code = NotFound desc = could not find container \"c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c\": container with ID starting with c6a2388355668ebeb37daadadeac87bff860d94bfa981e6499247be33739a04c not found: ID does not exist" Nov 03 17:53:24.047218 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.047169 4885 scope.go:122] "RemoveContainer" containerID="3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554" Nov 03 17:53:24.055322 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.055302 4885 scope.go:122] "RemoveContainer" containerID="3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554" Nov 03 17:53:24.055549 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:24.055530 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554\": container with ID starting with 3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554 not found: ID does not exist" containerID="3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554" Nov 03 17:53:24.055609 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.055555 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554"} err="failed to get container status \"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554\": rpc error: code = NotFound desc = could not find container \"3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554\": container with ID starting with 3b3a56b82794ae578ee0e63ee9a690784eeccf301ebe866a4ae11601ba33d554 not found: ID does not exist" Nov 03 17:53:24.250958 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.250918 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:53:24.256328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.256288 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-d8bl7"] Nov 03 17:53:24.370426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.370397 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:53:24.374126 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:24.374105 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e095fe-openshift-default-59765cdb87-p5qdw"] Nov 03 17:53:25.660730 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:25.660701 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801aa294-d130-4e9d-9188-09017008adb1" path="/var/lib/kubelet/pods/801aa294-d130-4e9d-9188-09017008adb1/volumes" Nov 03 17:53:25.661161 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:25.661146 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" path="/var/lib/kubelet/pods/dae2d7f2-46e2-4d35-a63c-ff283fb5a346/volumes" Nov 03 17:53:37.176829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.176785 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" containerID="cri-o://0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2" gracePeriod=45 Nov 03 17:53:37.202700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.202663 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused" Nov 03 17:53:37.207847 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.207801 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" containerID="cri-o://518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb" gracePeriod=45 Nov 03 17:53:37.509707 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.509685 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:53:37.515823 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.515798 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:53:37.612862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.612824 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.612875 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.612908 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.612946 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.612974 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\" (UniqueName: \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613043 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613010 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613081 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613115 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613153 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\") pod \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\" (UID: \"0926d730-d2c7-4feb-8c4d-eb1fac4943ac\") " Nov 03 17:53:37.613334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613107 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.613334 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613320 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.613567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613346 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\" (UniqueName: \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613394 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613447 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\") pod \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\" (UID: \"3c8d9cc2-5208-424b-8008-ee4e84c9821c\") " Nov 03 17:53:37.613567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613466 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.613858 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613711 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.613932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613830 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.613932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613893 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.613932 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.613909 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.614207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.614148 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.614331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.614235 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.615454 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.615426 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.615552 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.615443 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:37.615937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.615898 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:37.616051 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.616031 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9" pod "0926d730-d2c7-4feb-8c4d-eb1fac4943ac" (UID: "0926d730-d2c7-4feb-8c4d-eb1fac4943ac"). InnerVolumeSpecName "kube-api-access-nc4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:37.616337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.616319 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:37.616395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.616346 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7" pod "3c8d9cc2-5208-424b-8008-ee4e84c9821c" (UID: "3c8d9cc2-5208-424b-8008-ee4e84c9821c"). InnerVolumeSpecName "kube-api-access-jrcs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715507 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715536 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c8d9cc2-5208-424b-8008-ee4e84c9821c-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715548 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715559 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrcs7\" (UniqueName: \"kubernetes.io/projected/3c8d9cc2-5208-424b-8008-ee4e84c9821c-kube-api-access-jrcs7\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715573 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715585 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9cc2-5208-424b-8008-ee4e84c9821c-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715598 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715610 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc4n9\" (UniqueName: \"kubernetes.io/projected/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-kube-api-access-nc4n9\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:37.715639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:37.715621 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0926d730-d2c7-4feb-8c4d-eb1fac4943ac-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:38.126184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.124943 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129689 4885 cpu_manager.go:414] "containerMap: removing container" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129717 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129734 4885 cpu_manager.go:414] "containerMap: removing container" podUID="1b386b98-caad-4149-965c-f0e22b3ed4fc" containerName="maas-api" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129741 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b386b98-caad-4149-965c-f0e22b3ed4fc" containerName="maas-api" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129766 4885 cpu_manager.go:414] "containerMap: removing container" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129773 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129793 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129799 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129813 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129819 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129898 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerName="main" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129912 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a9d20c-c72b-4e0d-946a-2b7abd2d02e5" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129920 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae2d7f2-46e2-4d35-a63c-ff283fb5a346" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129929 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b386b98-caad-4149-965c-f0e22b3ed4fc" containerName="maas-api" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129938 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="801aa294-d130-4e9d-9188-09017008adb1" containerName="istio-proxy" Nov 03 17:53:38.135095 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.129946 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerName="main" Nov 03 17:53:38.137123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.137098 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.160170 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.160130 4885 generic.go:356] "Generic (PLEG): container finished" podID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" containerID="518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb" exitCode=0 Nov 03 17:53:38.160678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.160300 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" event={"ID":"3c8d9cc2-5208-424b-8008-ee4e84c9821c","Type":"ContainerDied","Data":"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb"} Nov 03 17:53:38.160678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.160345 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" event={"ID":"3c8d9cc2-5208-424b-8008-ee4e84c9821c","Type":"ContainerDied","Data":"272d5be4436b21ef544ca950d83f9c1b0e4955037f8db072976ee952509b710f"} Nov 03 17:53:38.160678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.160379 4885 scope.go:122] "RemoveContainer" containerID="518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb" Nov 03 17:53:38.160678 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.160625 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6" Nov 03 17:53:38.163739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.163715 4885 generic.go:356] "Generic (PLEG): container finished" podID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" containerID="0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2" exitCode=0 Nov 03 17:53:38.163909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.163890 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" Nov 03 17:53:38.164124 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.164103 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" event={"ID":"0926d730-d2c7-4feb-8c4d-eb1fac4943ac","Type":"ContainerDied","Data":"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2"} Nov 03 17:53:38.164250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.164227 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv" event={"ID":"0926d730-d2c7-4feb-8c4d-eb1fac4943ac","Type":"ContainerDied","Data":"6e9797c480304fd6ea7813c8eb9db7e9598b59c55d1cce8661d56fb07eae6fee"} Nov 03 17:53:38.173452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.173422 4885 scope.go:122] "RemoveContainer" containerID="518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb" Nov 03 17:53:38.173749 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:38.173731 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb\": container with ID starting with 518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb not found: ID does not exist" containerID="518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb" Nov 03 17:53:38.173823 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.173755 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb"} err="failed to get container status \"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb\": rpc error: code = NotFound desc = could not find container \"518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb\": container with ID starting with 518fd8bd222d142ccac7e6f795bf5c5e0c2330a07378d0ef3731a31200d31bfb not found: ID does not exist" Nov 03 17:53:38.173823 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.173773 4885 scope.go:122] "RemoveContainer" containerID="0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2" Nov 03 17:53:38.183137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.182937 4885 scope.go:122] "RemoveContainer" containerID="0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2" Nov 03 17:53:38.183396 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:38.183303 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2\": container with ID starting with 0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2 not found: ID does not exist" containerID="0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2" Nov 03 17:53:38.183396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.183344 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2"} err="failed to get container status \"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2\": rpc error: code = NotFound desc = could not find container \"0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2\": container with ID starting with 0dd293760f5f0df7574c3e577301729d02978e359578105a0ce12a296eb27ec2 not found: ID does not exist" Nov 03 17:53:38.187289 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.187261 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:38.191436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191237 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\"" Nov 03 17:53:38.191436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191281 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\"" Nov 03 17:53:38.191436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191286 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\"" Nov 03 17:53:38.191436 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191364 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\"" Nov 03 17:53:38.191645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191440 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\"" Nov 03 17:53:38.191645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191491 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\"" Nov 03 17:53:38.191645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191577 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\"" Nov 03 17:53:38.191645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.191623 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\"" Nov 03 17:53:38.221112 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.221055 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.221112 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.221123 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6p59\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.221294 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.221211 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.221294 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.221253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.322074 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322040 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.322253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322100 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6p59\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.322253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322132 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.322253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322158 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.322845 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322788 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.323037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.322905 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.324914 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.324895 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.373459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.373434 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:53:38.378538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.378467 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/premium-w3-2355f4-kserve-5bcf4d5bf4-6wtpv"] Nov 03 17:53:38.385725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.385699 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6p59\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.450378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.450349 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:38.495998 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.495966 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:53:38.499029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.499000 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2355f4/simulator-w3-2355f4-kserve-76bf65b475-w2rf6"] Nov 03 17:53:38.821973 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:53:38.821945 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7f1fb0_cc9e_4839_9c0f_31fe09756d1a.slice/crio-ff0b28873a9fa124b78e91ada9c19925a8fc1b35eeaf8a27bcae83e43a6d571e WatchSource:0}: Error finding container ff0b28873a9fa124b78e91ada9c19925a8fc1b35eeaf8a27bcae83e43a6d571e: Status 404 returned error can't find the container with id ff0b28873a9fa124b78e91ada9c19925a8fc1b35eeaf8a27bcae83e43a6d571e Nov 03 17:53:38.824268 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:38.824240 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:39.171483 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.171433 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" event={"ID":"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a","Type":"ContainerStarted","Data":"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a"} Nov 03 17:53:39.171653 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.171509 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" event={"ID":"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a","Type":"ContainerStarted","Data":"ff0b28873a9fa124b78e91ada9c19925a8fc1b35eeaf8a27bcae83e43a6d571e"} Nov 03 17:53:39.171653 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.171644 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:39.293521 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.293470 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" podStartSLOduration=1.293455408 podStartE2EDuration="1.293455408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:39.29182676 +0000 UTC m=+2161.746515783" watchObservedRunningTime="2026-11-03 17:53:39.293455408 +0000 UTC m=+2161.748144477" Nov 03 17:53:39.664046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.664013 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0926d730-d2c7-4feb-8c4d-eb1fac4943ac" path="/var/lib/kubelet/pods/0926d730-d2c7-4feb-8c4d-eb1fac4943ac/volumes" Nov 03 17:53:39.664441 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:39.664427 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8d9cc2-5208-424b-8008-ee4e84c9821c" path="/var/lib/kubelet/pods/3c8d9cc2-5208-424b-8008-ee4e84c9821c/volumes" Nov 03 17:53:45.620666 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.620631 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:53:45.624294 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.624274 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.679518 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.679486 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:53:45.706578 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706550 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.706733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706604 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjsk\" (UniqueName: \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.706733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706684 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.706804 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706744 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.706837 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706802 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.706869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.706838 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.728012 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.727987 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fe8fde\"/\"explicit-model-fe8fde-kserve-self-signed-certs\"" Nov 03 17:53:45.728187 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.728049 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fe8fde\"/\"openshift-service-ca.crt\"" Nov 03 17:53:45.732376 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.732361 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fe8fde\"/\"kube-root-ca.crt\"" Nov 03 17:53:45.732549 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.732535 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fe8fde\"/\"default-dockercfg-97tvs\"" Nov 03 17:53:45.808241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808205 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.808241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808257 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.808645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808333 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.808645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808452 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.808645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808502 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjsk\" (UniqueName: \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.808645 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.808531 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.809676 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.809644 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.810375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.810349 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.810606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.810585 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.812030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.812002 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.812763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.812740 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.873032 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.872960 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjsk\" (UniqueName: \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\") pod \"explicit-model-fe8fde-kserve-7d9db5799f-qldvq\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:45.878401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.878373 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:53:45.944584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:45.944543 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:46.324429 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:53:46.324398 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90835e6c_9466_4792_a5e2_4266871e27e9.slice/crio-32fdcfd98f97a390d4943d2dd6dfd95762f793a4792918514768ab8e14bd4583 WatchSource:0}: Error finding container 32fdcfd98f97a390d4943d2dd6dfd95762f793a4792918514768ab8e14bd4583: Status 404 returned error can't find the container with id 32fdcfd98f97a390d4943d2dd6dfd95762f793a4792918514768ab8e14bd4583 Nov 03 17:53:46.326058 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:46.326035 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:53:47.214823 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:47.214787 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" event={"ID":"90835e6c-9466-4792-a5e2-4266871e27e9","Type":"ContainerStarted","Data":"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20"} Nov 03 17:53:47.215355 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:47.214835 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" event={"ID":"90835e6c-9466-4792-a5e2-4266871e27e9","Type":"ContainerStarted","Data":"32fdcfd98f97a390d4943d2dd6dfd95762f793a4792918514768ab8e14bd4583"} Nov 03 17:53:47.337405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:47.337347 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" podStartSLOduration=2.337329877 podStartE2EDuration="2.337329877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:47.336225765 +0000 UTC m=+2169.790914789" watchObservedRunningTime="2026-11-03 17:53:47.337329877 +0000 UTC m=+2169.792018902" Nov 03 17:53:47.945739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:47.945700 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:48.605671 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:48.605628 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:53:48.841495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:48.841466 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:48.844551 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:48.844518 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:53:49.503923 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:49.503885 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:53:49.504165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:49.504144 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" containerID="cri-o://c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff" gracePeriod=30 Nov 03 17:53:53.842641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:53.842590 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerName="payload-processing" containerID="cri-o://869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a" gracePeriod=25 Nov 03 17:53:54.133516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.133490 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:54.193895 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.193870 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\") pod \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " Nov 03 17:53:54.194056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.193913 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\") pod \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " Nov 03 17:53:54.194056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194015 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\") pod \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " Nov 03 17:53:54.194225 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194090 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\") pod \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\" (UID: \"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a\") " Nov 03 17:53:54.194279 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194244 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume" pod "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" (UID: "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:54.194407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194377 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume" pod "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" (UID: "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:54.194514 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194500 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.194572 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.194517 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.196331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.196306 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59" pod "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" (UID: "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a"). InnerVolumeSpecName "kube-api-access-h6p59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:54.196412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.196338 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials" pod "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" (UID: "6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:54.248207 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.248181 4885 generic.go:356] "Generic (PLEG): container finished" podID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerID="869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a" exitCode=0 Nov 03 17:53:54.248368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.248230 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" Nov 03 17:53:54.248368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.248252 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" event={"ID":"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a","Type":"ContainerDied","Data":"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a"} Nov 03 17:53:54.248368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.248296 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r" event={"ID":"6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a","Type":"ContainerDied","Data":"ff0b28873a9fa124b78e91ada9c19925a8fc1b35eeaf8a27bcae83e43a6d571e"} Nov 03 17:53:54.248368 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.248330 4885 scope.go:122] "RemoveContainer" containerID="869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a" Nov 03 17:53:54.259581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.259565 4885 scope.go:122] "RemoveContainer" containerID="869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a" Nov 03 17:53:54.260012 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:54.259962 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a\": container with ID starting with 869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a not found: ID does not exist" containerID="869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a" Nov 03 17:53:54.260012 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.260001 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a"} err="failed to get container status \"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a\": rpc error: code = NotFound desc = could not find container \"869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a\": container with ID starting with 869fd93af9363cfd5dc032910cb9d294cffb6764dd21a8c5529798e955eb970a not found: ID does not exist" Nov 03 17:53:54.295312 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.295281 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.295457 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.295315 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6p59\" (UniqueName: \"kubernetes.io/projected/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a-kube-api-access-h6p59\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.380796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.380760 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:54.393197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.393169 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8f67x4r"] Nov 03 17:53:54.770351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.770325 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:53:54.901574 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901492 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901578 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901613 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901653 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901686 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901773 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901802 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901812 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901849 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.901910 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\") pod \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\" (UID: \"7eabb1a3-fcc8-47c0-bb9d-133c8844d483\") " Nov 03 17:53:54.902019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902005 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:54.902626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902179 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:54.902626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902314 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.902626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902323 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:54.902626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902333 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.902626 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902348 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.902872 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.902659 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:54.905877 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.905847 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "kube-api-access-xtcrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:54.906397 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.906374 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:53:54.910967 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.910936 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:54.919104 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:54.915214 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token" pod "7eabb1a3-fcc8-47c0-bb9d-133c8844d483" (UID: "7eabb1a3-fcc8-47c0-bb9d-133c8844d483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:55.003641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003607 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.003641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003637 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.003641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003647 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.003953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003657 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.003953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003666 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.003953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.003680 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtcrv\" (UniqueName: \"kubernetes.io/projected/7eabb1a3-fcc8-47c0-bb9d-133c8844d483-kube-api-access-xtcrv\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:55.254699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.254614 4885 generic.go:356] "Generic (PLEG): container finished" podID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerID="c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff" exitCode=0 Nov 03 17:53:55.254699 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.254686 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" Nov 03 17:53:55.254897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.254700 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" event={"ID":"7eabb1a3-fcc8-47c0-bb9d-133c8844d483","Type":"ContainerDied","Data":"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff"} Nov 03 17:53:55.254897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.254797 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c" event={"ID":"7eabb1a3-fcc8-47c0-bb9d-133c8844d483","Type":"ContainerDied","Data":"7530b5c433859dffe86619291e88dc4a39f8825c22c6670442bf6e25372a16fc"} Nov 03 17:53:55.254897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.254830 4885 scope.go:122] "RemoveContainer" containerID="c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff" Nov 03 17:53:55.264041 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.264020 4885 scope.go:122] "RemoveContainer" containerID="c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff" Nov 03 17:53:55.264337 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:53:55.264312 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff\": container with ID starting with c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff not found: ID does not exist" containerID="c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff" Nov 03 17:53:55.264383 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.264352 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff"} err="failed to get container status \"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff\": rpc error: code = NotFound desc = could not find container \"c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff\": container with ID starting with c69034083d63d9e0c7b589f11e94873f37b82d979814d41cdfa662516d1c1bff not found: ID does not exist" Nov 03 17:53:55.431464 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.431427 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:53:55.437831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.437808 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-cv87c"] Nov 03 17:53:55.662097 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.662052 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" path="/var/lib/kubelet/pods/6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a/volumes" Nov 03 17:53:55.662496 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.662483 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" path="/var/lib/kubelet/pods/7eabb1a3-fcc8-47c0-bb9d-133c8844d483/volumes" Nov 03 17:53:55.945136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.945039 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:55.959237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:55.959211 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:53:56.279724 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:53:56.279635 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:54:08.070262 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:08.070182 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:54:09.493833 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:09.493798 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:54:18.604703 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:18.604665 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:54:24.513412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.513364 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" podUID="90835e6c-9466-4792-a5e2-4266871e27e9" containerName="main" containerID="cri-o://d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20" gracePeriod=45 Nov 03 17:54:24.765343 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.765282 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:54:24.833021 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.832993 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833031 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\" (UniqueName: \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833054 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833110 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833152 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833191 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833188 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\") pod \"90835e6c-9466-4792-a5e2-4266871e27e9\" (UID: \"90835e6c-9466-4792-a5e2-4266871e27e9\") " Nov 03 17:54:24.833474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833416 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:24.833474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833431 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:24.833592 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.833506 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:24.835493 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.835473 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:24.835753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.835727 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "kube-api-access-4tjsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:24.835753 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.835732 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm" pod "90835e6c-9466-4792-a5e2-4266871e27e9" (UID: "90835e6c-9466-4792-a5e2-4266871e27e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:24.934345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934313 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:24.934345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934341 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tjsk\" (UniqueName: \"kubernetes.io/projected/90835e6c-9466-4792-a5e2-4266871e27e9-kube-api-access-4tjsk\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:24.934529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934353 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:24.934529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934363 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90835e6c-9466-4792-a5e2-4266871e27e9-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:24.934529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934372 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:24.934529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:24.934379 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90835e6c-9466-4792-a5e2-4266871e27e9-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:25.427030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.426992 4885 generic.go:356] "Generic (PLEG): container finished" podID="90835e6c-9466-4792-a5e2-4266871e27e9" containerID="d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20" exitCode=0 Nov 03 17:54:25.427220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.427036 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" Nov 03 17:54:25.427220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.427044 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" event={"ID":"90835e6c-9466-4792-a5e2-4266871e27e9","Type":"ContainerDied","Data":"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20"} Nov 03 17:54:25.427220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.427102 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq" event={"ID":"90835e6c-9466-4792-a5e2-4266871e27e9","Type":"ContainerDied","Data":"32fdcfd98f97a390d4943d2dd6dfd95762f793a4792918514768ab8e14bd4583"} Nov 03 17:54:25.427220 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.427119 4885 scope.go:122] "RemoveContainer" containerID="d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20" Nov 03 17:54:25.435642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.435624 4885 scope.go:122] "RemoveContainer" containerID="d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20" Nov 03 17:54:25.435881 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:25.435860 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20\": container with ID starting with d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20 not found: ID does not exist" containerID="d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20" Nov 03 17:54:25.435931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.435892 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20"} err="failed to get container status \"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20\": rpc error: code = NotFound desc = could not find container \"d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20\": container with ID starting with d513bf575fb88c2fac55e26f3171873a8118c3ac6eb339d7ea6d19690629db20 not found: ID does not exist" Nov 03 17:54:25.595467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.595425 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:54:25.662328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:25.662301 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-fe8fde/explicit-model-fe8fde-kserve-7d9db5799f-qldvq"] Nov 03 17:54:27.661292 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:27.661258 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90835e6c-9466-4792-a5e2-4266871e27e9" path="/var/lib/kubelet/pods/90835e6c-9466-4792-a5e2-4266871e27e9/volumes" Nov 03 17:54:28.154870 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:28.154836 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:54:48.605009 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:48.604974 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:54:49.978056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:49.978022 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:54:49.978584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:49.978262 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" containerID="cri-o://befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d" gracePeriod=30 Nov 03 17:54:53.498774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:53.498738 4885 patch_prober.go:29] interesting pod/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:54:53.499168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:53.498784 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:54:54.980819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:54.980779 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:54:54.981300 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:54.980973 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" containerID="cri-o://bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941" gracePeriod=30 Nov 03 17:54:55.236135 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.236059 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:54:55.347270 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347234 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347280 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347301 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347342 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347369 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347450 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347417 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347454 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347505 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347529 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:55.347703 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347562 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\") pod \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\" (UID: \"f4aef24f-6e2e-4c78-bb99-7a2614ea900b\") " Nov 03 17:54:55.347909 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347862 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:55.347968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347909 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:55.347968 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.347918 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.348113 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.348046 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:55.348182 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.348136 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:55.349759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.349729 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:55.349759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.349736 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:54:55.350204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.350181 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "kube-api-access-5k5xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:55.350204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.350193 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token" pod "f4aef24f-6e2e-4c78-bb99-7a2614ea900b" (UID: "f4aef24f-6e2e-4c78-bb99-7a2614ea900b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:55.448614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448584 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448614 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448611 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5xb\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-kube-api-access-5k5xb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448622 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448631 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448639 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448647 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448655 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.448812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.448664 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4aef24f-6e2e-4c78-bb99-7a2614ea900b-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:55.578679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.578588 4885 generic.go:356] "Generic (PLEG): container finished" podID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerID="befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d" exitCode=0 Nov 03 17:54:55.578679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.578655 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" Nov 03 17:54:55.578870 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.578686 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" event={"ID":"f4aef24f-6e2e-4c78-bb99-7a2614ea900b","Type":"ContainerDied","Data":"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d"} Nov 03 17:54:55.578870 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.578749 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g" event={"ID":"f4aef24f-6e2e-4c78-bb99-7a2614ea900b","Type":"ContainerDied","Data":"977e05f53ea81f9ba7d08feb87408945c87b7a8a7439745ea30299774b0814c9"} Nov 03 17:54:55.578870 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.578773 4885 scope.go:122] "RemoveContainer" containerID="befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d" Nov 03 17:54:55.587567 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.587553 4885 scope.go:122] "RemoveContainer" containerID="befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d" Nov 03 17:54:55.587826 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:55.587812 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d\": container with ID starting with befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d not found: ID does not exist" containerID="befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d" Nov 03 17:54:55.587878 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.587833 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d"} err="failed to get container status \"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d\": rpc error: code = NotFound desc = could not find container \"befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d\": container with ID starting with befbe95913be210a4be3213b9d2dbf1424cd4b526a1ee9f4c4bd742d513d676d not found: ID does not exist" Nov 03 17:54:55.749082 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.749045 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:54:55.799294 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:55.799258 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-dbx8g"] Nov 03 17:54:57.227388 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.227358 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:54:57.227798 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.227564 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" podUID="bce56faf-4160-4880-bb6e-6776afa8370d" containerName="maas-api" containerID="cri-o://22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795" gracePeriod=30 Nov 03 17:54:57.510811 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.510780 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:54:57.572015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.571988 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\") pod \"bce56faf-4160-4880-bb6e-6776afa8370d\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " Nov 03 17:54:57.572204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.572055 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\" (UniqueName: \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\") pod \"bce56faf-4160-4880-bb6e-6776afa8370d\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " Nov 03 17:54:57.572204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.572146 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") pod \"bce56faf-4160-4880-bb6e-6776afa8370d\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " Nov 03 17:54:57.572204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.572191 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\") pod \"bce56faf-4160-4880-bb6e-6776afa8370d\" (UID: \"bce56faf-4160-4880-bb6e-6776afa8370d\") " Nov 03 17:54:57.572433 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.572408 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca" pod "bce56faf-4160-4880-bb6e-6776afa8370d" (UID: "bce56faf-4160-4880-bb6e-6776afa8370d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:57.572570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.572540 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce56faf-4160-4880-bb6e-6776afa8370d-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:57.574338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.574311 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls" pod "bce56faf-4160-4880-bb6e-6776afa8370d" (UID: "bce56faf-4160-4880-bb6e-6776afa8370d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:57.574338 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.574324 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb" pod "bce56faf-4160-4880-bb6e-6776afa8370d" (UID: "bce56faf-4160-4880-bb6e-6776afa8370d"). InnerVolumeSpecName "kube-api-access-vnzvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:57.574468 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.574450 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs" pod "bce56faf-4160-4880-bb6e-6776afa8370d" (UID: "bce56faf-4160-4880-bb6e-6776afa8370d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:57.599893 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.599857 4885 generic.go:356] "Generic (PLEG): container finished" podID="bce56faf-4160-4880-bb6e-6776afa8370d" containerID="22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795" exitCode=0 Nov 03 17:54:57.600108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.599938 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" event={"ID":"bce56faf-4160-4880-bb6e-6776afa8370d","Type":"ContainerDied","Data":"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795"} Nov 03 17:54:57.600108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.599989 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" event={"ID":"bce56faf-4160-4880-bb6e-6776afa8370d","Type":"ContainerDied","Data":"35f4fbe5186af350fa4ac0c9007592ea03d61d77067f8b09b5e881218be27a05"} Nov 03 17:54:57.600108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.600019 4885 scope.go:122] "RemoveContainer" containerID="22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795" Nov 03 17:54:57.600286 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.600180 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd" Nov 03 17:54:57.661287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.661257 4885 scope.go:122] "RemoveContainer" containerID="22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795" Nov 03 17:54:57.661654 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:57.661611 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795\": container with ID starting with 22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795 not found: ID does not exist" containerID="22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795" Nov 03 17:54:57.661728 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.661668 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795"} err="failed to get container status \"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795\": rpc error: code = NotFound desc = could not find container \"22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795\": container with ID starting with 22e25d78c2a7ed9a008dcd21567058b624626f438410bfb2f8e2da8fcc315795 not found: ID does not exist" Nov 03 17:54:57.667691 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.667667 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" path="/var/lib/kubelet/pods/f4aef24f-6e2e-4c78-bb99-7a2614ea900b/volumes" Nov 03 17:54:57.672993 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.672971 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzvb\" (UniqueName: \"kubernetes.io/projected/bce56faf-4160-4880-bb6e-6776afa8370d-kube-api-access-vnzvb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:57.673110 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.672995 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:57.673110 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.673005 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bce56faf-4160-4880-bb6e-6776afa8370d-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:54:57.692375 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:54:57.692353 4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce56faf_4160_4880_bb6e_6776afa8370d.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:54:57.805671 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.805594 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:54:57.813481 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:57.813451 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-fe8fde-6f5d954c54-q65wd"] Nov 03 17:54:59.661167 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:59.661136 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce56faf-4160-4880-bb6e-6776afa8370d" path="/var/lib/kubelet/pods/bce56faf-4160-4880-bb6e-6776afa8370d/volumes" Nov 03 17:54:59.733582 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:59.733544 4885 patch_prober.go:29] interesting pod/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:54:59.733768 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:54:59.733607 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:55:00.250600 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.250538 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:55:00.295825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.295793 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.295825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.295828 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.295865 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.295898 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.295942 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296017 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296055 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296133 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296165 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\") pod \"892b780d-778e-4cb4-9140-6cf014dab4c1\" (UID: \"892b780d-778e-4cb4-9140-6cf014dab4c1\") " Nov 03 17:55:00.296299 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296197 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:00.296513 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296438 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:00.296656 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296543 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.296774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296721 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.296774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296627 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:00.296774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296636 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:00.296916 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.296792 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:00.298249 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.298230 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "kube-api-access-95k4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:00.298328 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.298230 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:00.298640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.298625 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:00.298682 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.298656 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token" pod "892b780d-778e-4cb4-9140-6cf014dab4c1" (UID: "892b780d-778e-4cb4-9140-6cf014dab4c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:00.397528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397502 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95k4c\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-kube-api-access-95k4c\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397526 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397535 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397545 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397553 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397563 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892b780d-778e-4cb4-9140-6cf014dab4c1-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.397783 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.397572 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892b780d-778e-4cb4-9140-6cf014dab4c1-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:00.617405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.617170 4885 generic.go:356] "Generic (PLEG): container finished" podID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerID="bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941" exitCode=0 Nov 03 17:55:00.617405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.617235 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" Nov 03 17:55:00.617405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.617262 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" event={"ID":"892b780d-778e-4cb4-9140-6cf014dab4c1","Type":"ContainerDied","Data":"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941"} Nov 03 17:55:00.617405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.617315 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz" event={"ID":"892b780d-778e-4cb4-9140-6cf014dab4c1","Type":"ContainerDied","Data":"a0c6ec777532d9a448ef84a1300f6379e9ce793f7cb543d5004276a2c00da6b5"} Nov 03 17:55:00.617405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.617347 4885 scope.go:122] "RemoveContainer" containerID="bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941" Nov 03 17:55:00.637799 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.637776 4885 scope.go:122] "RemoveContainer" containerID="bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941" Nov 03 17:55:00.638178 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:00.638157 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941\": container with ID starting with bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941 not found: ID does not exist" containerID="bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941" Nov 03 17:55:00.638241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.638188 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941"} err="failed to get container status \"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941\": rpc error: code = NotFound desc = could not find container \"bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941\": container with ID starting with bcba4ac0ba59be29a5714f10efbf80f43c5e0a56a768ec0462cf220064875941 not found: ID does not exist" Nov 03 17:55:00.788091 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.787961 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:55:00.797094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:00.794976 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-6hxdz"] Nov 03 17:55:01.661402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.661360 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" path="/var/lib/kubelet/pods/892b780d-778e-4cb4-9140-6cf014dab4c1/volumes" Nov 03 17:55:01.854813 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.854770 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:01.855387 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855367 4885 cpu_manager.go:414] "containerMap: removing container" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerName="payload-processing" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855389 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerName="payload-processing" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855410 4885 cpu_manager.go:414] "containerMap: removing container" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855417 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855434 4885 cpu_manager.go:414] "containerMap: removing container" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855441 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855454 4885 cpu_manager.go:414] "containerMap: removing container" podUID="90835e6c-9466-4792-a5e2-4266871e27e9" containerName="main" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855461 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90835e6c-9466-4792-a5e2-4266871e27e9" containerName="main" Nov 03 17:55:01.855472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855473 4885 cpu_manager.go:414] "containerMap: removing container" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855479 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855510 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bce56faf-4160-4880-bb6e-6776afa8370d" containerName="maas-api" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855518 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce56faf-4160-4880-bb6e-6776afa8370d" containerName="maas-api" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855605 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce56faf-4160-4880-bb6e-6776afa8370d" containerName="maas-api" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855620 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7f1fb0-cc9e-4839-9c0f-31fe09756d1a" containerName="payload-processing" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855629 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eabb1a3-fcc8-47c0-bb9d-133c8844d483" containerName="istio-proxy" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855639 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="892b780d-778e-4cb4-9140-6cf014dab4c1" containerName="istio-proxy" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855648 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="90835e6c-9466-4792-a5e2-4266871e27e9" containerName="main" Nov 03 17:55:01.855851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.855656 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4aef24f-6e2e-4c78-bb99-7a2614ea900b" containerName="istio-proxy" Nov 03 17:55:01.860505 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.860484 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949412 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949473 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949534 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949640 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vrn\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949668 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949693 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949759 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949786 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.953900 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.949818 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:01.954528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.954424 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:01.968831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:01.968793 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-909de6-openshift-default-dockercfg-j4wrv\"" Nov 03 17:55:02.050701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050669 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050701 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050699 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050721 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050739 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050760 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050788 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050836 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vrn\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050852 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.050955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.050872 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.051344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.051168 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.051344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.051320 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.051344 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.051334 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.051565 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.051539 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.051641 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.051545 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.053183 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.053162 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.053640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.053619 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.114589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.114557 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vrn\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.114737 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.114709 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\") pod \"e2e-noten-909de6-openshift-default-5947fc965f-dd4nv\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.173015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.172985 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:02.592234 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:55:02.592202 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce034f7_9564_43ee_a2f4_90f8ca8578b4.slice/crio-ff348e6e83a1623b034d30392b149f3459807073455a71111fb7348afe8322cd WatchSource:0}: Error finding container ff348e6e83a1623b034d30392b149f3459807073455a71111fb7348afe8322cd: Status 404 returned error can't find the container with id ff348e6e83a1623b034d30392b149f3459807073455a71111fb7348afe8322cd Nov 03 17:55:02.594120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.594095 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:02.594352 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.594324 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:02.594395 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.594384 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:02.594424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.594413 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:02.629722 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:02.629687 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" event={"ID":"2ce034f7-9564-43ee-a2f4-90f8ca8578b4","Type":"ContainerStarted","Data":"ff348e6e83a1623b034d30392b149f3459807073455a71111fb7348afe8322cd"} Nov 03 17:55:03.636284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:03.636244 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" event={"ID":"2ce034f7-9564-43ee-a2f4-90f8ca8578b4","Type":"ContainerStarted","Data":"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc"} Nov 03 17:55:03.812032 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:03.811980 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" podStartSLOduration=2.811964485 podStartE2EDuration="2.811964485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:03.810397871 +0000 UTC m=+2246.265086894" watchObservedRunningTime="2026-11-03 17:55:03.811964485 +0000 UTC m=+2246.266653505" Nov 03 17:55:04.174140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:04.174109 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:04.179320 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:04.179296 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:04.652298 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:04.652269 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:04.653899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:04.653878 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:04.744760 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:04.744727 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:55:08.614163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.614129 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:08.617786 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.617768 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.673706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.673678 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:08.711448 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711421 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.711700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711678 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.711785 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711718 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.711785 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711768 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.711879 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711807 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.711980 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.711968 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj55j\" (UniqueName: \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.723329 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.723301 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-909de6\"/\"kube-root-ca.crt\"" Nov 03 17:55:08.723329 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.723302 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-909de6\"/\"openshift-service-ca.crt\"" Nov 03 17:55:08.723885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.723868 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-909de6\"/\"default-dockercfg-s7m55\"" Nov 03 17:55:08.723885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.723874 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-909de6\"/\"noten-model-909de6-kserve-self-signed-certs\"" Nov 03 17:55:08.812473 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812440 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812491 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812539 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj55j\" (UniqueName: \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812590 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812633 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812670 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.812978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.812951 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.813060 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.813042 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.813255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.813231 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.815121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.815094 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.815519 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.815502 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.881476 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.881444 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj55j\" (UniqueName: \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\") pod \"noten-model-909de6-kserve-7465c4c76f-xpz2b\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:08.946093 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:08.946037 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:09.315800 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:55:09.315770 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb99382e_9c30_4ee6_996f_2fdcb449f2ba.slice/crio-e6ac4dbfda4c5b3e9f2d66ef23c18b5fdf169e7e46170de7f5ccc45104ff82f3 WatchSource:0}: Error finding container e6ac4dbfda4c5b3e9f2d66ef23c18b5fdf169e7e46170de7f5ccc45104ff82f3: Status 404 returned error can't find the container with id e6ac4dbfda4c5b3e9f2d66ef23c18b5fdf169e7e46170de7f5ccc45104ff82f3 Nov 03 17:55:09.318257 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:09.318154 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:09.677233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:09.677198 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" event={"ID":"eb99382e-9c30-4ee6-996f-2fdcb449f2ba","Type":"ContainerStarted","Data":"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e"} Nov 03 17:55:09.677233 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:09.677239 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" event={"ID":"eb99382e-9c30-4ee6-996f-2fdcb449f2ba","Type":"ContainerStarted","Data":"e6ac4dbfda4c5b3e9f2d66ef23c18b5fdf169e7e46170de7f5ccc45104ff82f3"} Nov 03 17:55:09.797046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:09.796992 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" podStartSLOduration=1.796978438 podStartE2EDuration="1.796978438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:09.796830151 +0000 UTC m=+2252.251519166" watchObservedRunningTime="2026-11-03 17:55:09.796978438 +0000 UTC m=+2252.251667461" Nov 03 17:55:09.946534 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:09.946452 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:10.165796 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.165761 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:55:10.166044 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.166009 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" containerID="cri-o://9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7" gracePeriod=30 Nov 03 17:55:10.216250 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.216151 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:55:10.216443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.216413 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" containerID="cri-o://1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d" gracePeriod=30 Nov 03 17:55:10.217844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.217819 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:55:10.218123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.218090 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" containerID="cri-o://2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d" gracePeriod=30 Nov 03 17:55:10.220035 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.220012 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:55:10.220427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:10.220399 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" containerID="cri-o://c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609" gracePeriod=30 Nov 03 17:55:13.803480 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:13.803436 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-rt6t4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:55:13.805700 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:13.803491 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:55:15.377853 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.377811 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-n5wkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:15.378345 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.377895 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 17:55:15.474667 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.474638 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:55:15.535819 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.535789 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:55:15.546874 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.546848 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:55:15.560438 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.560415 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:55:15.600596 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600569 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600608 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600643 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600684 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600711 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600752 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600777 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600816 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600845 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600872 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600897 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600932 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600968 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601009 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.600995 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601026 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601091 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601133 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601272 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601372 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601604 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601619 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601658 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601714 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601768 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.601809 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601794 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601820 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601848 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601886 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601886 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601935 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.601985 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602020 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602047 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\") pod \"76a06ce6-651d-461f-8631-c9dfedce328b\" (UID: \"76a06ce6-651d-461f-8631-c9dfedce328b\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602103 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602138 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602165 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602217 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.602427 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602247 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\") pod \"98023514-7416-4f9b-a29d-cff2720447d3\" (UID: \"98023514-7416-4f9b-a29d-cff2720447d3\") " Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602443 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602473 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\") pod \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\" (UID: \"7cd69a74-705a-4c8e-bf13-ff73aec9f58f\") " Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602507 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\") pod \"619c15b7-5793-4589-81b5-1a92f2c64741\" (UID: \"619c15b7-5793-4589-81b5-1a92f2c64741\") " Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602909 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602929 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602943 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619c15b7-5793-4589-81b5-1a92f2c64741-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602957 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.602971 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.603189 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.603515 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.603397 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.604040 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.603785 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.604040 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.603988 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.604190 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.604430 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.604870 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "kube-api-access-5wftl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.604948 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.605741 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.606122 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.606339 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607173 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607227 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607373 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607444 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "kube-api-access-5w725". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607500 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607660 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607680 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607830 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.607866 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.608002 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.608493 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.608503 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.608663 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.609818 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.610470 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo" pod "98023514-7416-4f9b-a29d-cff2720447d3" (UID: "98023514-7416-4f9b-a29d-cff2720447d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.610664 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w" pod "619c15b7-5793-4589-81b5-1a92f2c64741" (UID: "619c15b7-5793-4589-81b5-1a92f2c64741"). InnerVolumeSpecName "kube-api-access-9h84w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.611003 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.611765 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:15.612263 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.612021 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk" pod "76a06ce6-651d-461f-8631-c9dfedce328b" (UID: "76a06ce6-651d-461f-8631-c9dfedce328b"). InnerVolumeSpecName "kube-api-access-t74xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.613587 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.612539 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token" pod "7cd69a74-705a-4c8e-bf13-ff73aec9f58f" (UID: "7cd69a74-705a-4c8e-bf13-ff73aec9f58f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:15.703498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703465 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h84w\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-kube-api-access-9h84w\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703498 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619c15b7-5793-4589-81b5-1a92f2c64741-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703511 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703525 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703535 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703547 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703560 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703571 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703582 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703594 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703606 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703619 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74xk\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-kube-api-access-t74xk\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703632 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703644 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76a06ce6-651d-461f-8631-c9dfedce328b-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703658 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703670 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98023514-7416-4f9b-a29d-cff2720447d3-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703681 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703693 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703704 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703718 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98023514-7416-4f9b-a29d-cff2720447d3-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703730 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703745 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703759 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.703769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703773 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619c15b7-5793-4589-81b5-1a92f2c64741-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703785 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76a06ce6-651d-461f-8631-c9dfedce328b-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703798 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wftl\" (UniqueName: \"kubernetes.io/projected/7cd69a74-705a-4c8e-bf13-ff73aec9f58f-kube-api-access-5wftl\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703813 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619c15b7-5793-4589-81b5-1a92f2c64741-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703828 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98023514-7416-4f9b-a29d-cff2720447d3-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703841 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76a06ce6-651d-461f-8631-c9dfedce328b-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703854 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76a06ce6-651d-461f-8631-c9dfedce328b-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.704606 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.703868 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w725\" (UniqueName: \"kubernetes.io/projected/98023514-7416-4f9b-a29d-cff2720447d3-kube-api-access-5w725\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:15.706875 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.706837 4885 generic.go:356] "Generic (PLEG): container finished" podID="98023514-7416-4f9b-a29d-cff2720447d3" containerID="c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609" exitCode=0 Nov 03 17:55:15.707034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.706879 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" Nov 03 17:55:15.707034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.706930 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" event={"ID":"98023514-7416-4f9b-a29d-cff2720447d3","Type":"ContainerDied","Data":"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609"} Nov 03 17:55:15.707034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.707010 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm" event={"ID":"98023514-7416-4f9b-a29d-cff2720447d3","Type":"ContainerDied","Data":"45842210271c418d012fb24600954bf0cd023ae25dd19dae36c26148844d8e69"} Nov 03 17:55:15.707283 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.707058 4885 scope.go:122] "RemoveContainer" containerID="c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609" Nov 03 17:55:15.708781 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.708756 4885 generic.go:356] "Generic (PLEG): container finished" podID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerID="1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d" exitCode=0 Nov 03 17:55:15.708899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.708806 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" Nov 03 17:55:15.708899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.708851 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" event={"ID":"7cd69a74-705a-4c8e-bf13-ff73aec9f58f","Type":"ContainerDied","Data":"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d"} Nov 03 17:55:15.709013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.708915 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4" event={"ID":"7cd69a74-705a-4c8e-bf13-ff73aec9f58f","Type":"ContainerDied","Data":"53735f91125ceb666a51d2c6f5e639c818add40a119e49ab2d470eb5bfe6f119"} Nov 03 17:55:15.710831 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.710808 4885 generic.go:356] "Generic (PLEG): container finished" podID="76a06ce6-651d-461f-8631-c9dfedce328b" containerID="9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7" exitCode=0 Nov 03 17:55:15.710943 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.710843 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" Nov 03 17:55:15.711029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.710994 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" event={"ID":"76a06ce6-651d-461f-8631-c9dfedce328b","Type":"ContainerDied","Data":"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7"} Nov 03 17:55:15.711120 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.711040 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc" event={"ID":"76a06ce6-651d-461f-8631-c9dfedce328b","Type":"ContainerDied","Data":"a5a1543f85dbf8d6a8aa647b6378414543a562e9e3c83dcbf44bcb232d021590"} Nov 03 17:55:15.713053 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.713030 4885 generic.go:356] "Generic (PLEG): container finished" podID="619c15b7-5793-4589-81b5-1a92f2c64741" containerID="2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d" exitCode=0 Nov 03 17:55:15.713175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.713101 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" Nov 03 17:55:15.713175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.713152 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" event={"ID":"619c15b7-5793-4589-81b5-1a92f2c64741","Type":"ContainerDied","Data":"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d"} Nov 03 17:55:15.713276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.713189 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx" event={"ID":"619c15b7-5793-4589-81b5-1a92f2c64741","Type":"ContainerDied","Data":"1034dd1525a64dbeb315a05a400645ac1d30983d006e0480c939f0ed0213c3e5"} Nov 03 17:55:15.722205 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.722186 4885 scope.go:122] "RemoveContainer" containerID="c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609" Nov 03 17:55:15.722469 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:15.722451 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609\": container with ID starting with c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609 not found: ID does not exist" containerID="c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609" Nov 03 17:55:15.722510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.722477 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609"} err="failed to get container status \"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609\": rpc error: code = NotFound desc = could not find container \"c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609\": container with ID starting with c89de02cb59d85075a46920400dc83226e8dd0e7b95ae95919ec215f0f4f9609 not found: ID does not exist" Nov 03 17:55:15.722510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.722502 4885 scope.go:122] "RemoveContainer" containerID="1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d" Nov 03 17:55:15.731956 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.731939 4885 scope.go:122] "RemoveContainer" containerID="1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d" Nov 03 17:55:15.732296 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:15.732271 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d\": container with ID starting with 1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d not found: ID does not exist" containerID="1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d" Nov 03 17:55:15.732398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.732301 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d"} err="failed to get container status \"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d\": rpc error: code = NotFound desc = could not find container \"1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d\": container with ID starting with 1a32c47934cdfadd9e2fb6aa680f7ab9566ad044e6b99e2a949700cd1ba2419d not found: ID does not exist" Nov 03 17:55:15.732398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.732320 4885 scope.go:122] "RemoveContainer" containerID="9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7" Nov 03 17:55:15.740887 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.740868 4885 scope.go:122] "RemoveContainer" containerID="9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7" Nov 03 17:55:15.741201 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:15.741175 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7\": container with ID starting with 9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7 not found: ID does not exist" containerID="9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7" Nov 03 17:55:15.741272 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.741208 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7"} err="failed to get container status \"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7\": rpc error: code = NotFound desc = could not find container \"9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7\": container with ID starting with 9b46c729930fbda4e9a53ab4095b5722d7cab16b780e751fbae732561d9bfba7 not found: ID does not exist" Nov 03 17:55:15.741272 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.741230 4885 scope.go:122] "RemoveContainer" containerID="2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d" Nov 03 17:55:15.754845 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.754820 4885 scope.go:122] "RemoveContainer" containerID="2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d" Nov 03 17:55:15.755334 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:15.755306 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d\": container with ID starting with 2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d not found: ID does not exist" containerID="2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d" Nov 03 17:55:15.755453 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:15.755354 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d"} err="failed to get container status \"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d\": rpc error: code = NotFound desc = could not find container \"2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d\": container with ID starting with 2a438af889f657305c0cf7f4037be759762dac1401ac0e7593cf963ea0de892d not found: ID does not exist" Nov 03 17:55:16.096031 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.095949 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:55:16.099840 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.099813 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-n5wkm"] Nov 03 17:55:16.214265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.214228 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:55:16.217099 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.217077 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-xv8cx"] Nov 03 17:55:16.331407 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.331369 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:55:16.334661 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.334632 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-rt6t4"] Nov 03 17:55:16.448090 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.448040 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:55:16.453244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:16.453220 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-c6nsc"] Nov 03 17:55:17.662776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:17.662739 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" path="/var/lib/kubelet/pods/619c15b7-5793-4589-81b5-1a92f2c64741/volumes" Nov 03 17:55:17.663179 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:17.663160 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" path="/var/lib/kubelet/pods/76a06ce6-651d-461f-8631-c9dfedce328b/volumes" Nov 03 17:55:17.663516 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:17.663504 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" path="/var/lib/kubelet/pods/7cd69a74-705a-4c8e-bf13-ff73aec9f58f/volumes" Nov 03 17:55:17.664197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:17.664171 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98023514-7416-4f9b-a29d-cff2720447d3" path="/var/lib/kubelet/pods/98023514-7416-4f9b-a29d-cff2720447d3/volumes" Nov 03 17:55:18.604495 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:18.604457 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:55:18.946706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:18.946671 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:18.958795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:18.958771 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:19.746931 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:19.746899 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:25.953546 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:25.953517 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:26.130426 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:26.130396 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:26.130632 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:26.130608 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" podUID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" containerName="istio-proxy" containerID="cri-o://61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc" gracePeriod=30 Nov 03 17:55:31.379378 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.379352 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:31.463832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463750 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.463832 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463786 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463844 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463898 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463942 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.463979 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464022 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464057 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464055 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464455 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464155 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\") pod \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\" (UID: \"2ce034f7-9564-43ee-a2f4-90f8ca8578b4\") " Nov 03 17:55:31.464455 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464399 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:55:31.464683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464581 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:31.464683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464668 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:31.464937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464812 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:31.464937 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.464829 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:31.466222 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.466195 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "kube-api-access-w7vrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:31.466314 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.466251 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:55:31.466488 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.466470 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:31.466665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.466635 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token" pod "2ce034f7-9564-43ee-a2f4-90f8ca8578b4" (UID: "2ce034f7-9564-43ee-a2f4-90f8ca8578b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:31.565411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565377 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565405 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565419 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565431 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565443 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565455 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565466 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565478 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vrn\" (UniqueName: \"kubernetes.io/projected/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-kube-api-access-w7vrn\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.565599 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.565492 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce034f7-9564-43ee-a2f4-90f8ca8578b4-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:31.792651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.792573 4885 generic.go:356] "Generic (PLEG): container finished" podID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" containerID="61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc" exitCode=0 Nov 03 17:55:31.792651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.792620 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" Nov 03 17:55:31.792651 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.792643 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" event={"ID":"2ce034f7-9564-43ee-a2f4-90f8ca8578b4","Type":"ContainerDied","Data":"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc"} Nov 03 17:55:31.792896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.792686 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv" event={"ID":"2ce034f7-9564-43ee-a2f4-90f8ca8578b4","Type":"ContainerDied","Data":"ff348e6e83a1623b034d30392b149f3459807073455a71111fb7348afe8322cd"} Nov 03 17:55:31.792896 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.792705 4885 scope.go:122] "RemoveContainer" containerID="61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc" Nov 03 17:55:31.800867 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.800851 4885 scope.go:122] "RemoveContainer" containerID="61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc" Nov 03 17:55:31.801239 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:31.801209 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc\": container with ID starting with 61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc not found: ID does not exist" containerID="61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc" Nov 03 17:55:31.801337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.801249 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc"} err="failed to get container status \"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc\": rpc error: code = NotFound desc = could not find container \"61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc\": container with ID starting with 61d589fb048dfcfdbd8977cf2bd1aed471b253ae58e041b1ff5a992948f7aecc not found: ID does not exist" Nov 03 17:55:31.962002 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.961970 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:31.965939 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:31.965915 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-909de6-openshift-default-5947fc965f-dd4nv"] Nov 03 17:55:33.661103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:33.661050 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" path="/var/lib/kubelet/pods/2ce034f7-9564-43ee-a2f4-90f8ca8578b4/volumes" Nov 03 17:55:38.435168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:38.435134 4885 scope.go:122] "RemoveContainer" containerID="c277a5488115e05cbac651b756c5e5e110c4b1956e9c87014727c1725c1c4773" Nov 03 17:55:40.972735 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:40.972691 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" podUID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" containerName="main" containerID="cri-o://0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e" gracePeriod=45 Nov 03 17:55:41.225792 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.225732 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:41.350447 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350413 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.350589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350474 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.350589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350545 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.350589 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350570 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.350769 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350704 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:41.350830 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350778 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:41.350886 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350833 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\" (UniqueName: \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.350942 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.350911 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\") pod \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\" (UID: \"eb99382e-9c30-4ee6-996f-2fdcb449f2ba\") " Nov 03 17:55:41.351136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.351117 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:41.351375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.351199 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.351375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.351216 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.351375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.351232 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.352891 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.352864 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:55:41.352986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.352917 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "kube-api-access-cj55j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:55:41.353121 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.353106 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs" pod "eb99382e-9c30-4ee6-996f-2fdcb449f2ba" (UID: "eb99382e-9c30-4ee6-996f-2fdcb449f2ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:55:41.451639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.451606 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.451639 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.451633 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.451801 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.451644 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj55j\" (UniqueName: \"kubernetes.io/projected/eb99382e-9c30-4ee6-996f-2fdcb449f2ba-kube-api-access-cj55j\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:55:41.846241 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.846205 4885 generic.go:356] "Generic (PLEG): container finished" podID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" containerID="0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e" exitCode=0 Nov 03 17:55:41.846402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.846252 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" event={"ID":"eb99382e-9c30-4ee6-996f-2fdcb449f2ba","Type":"ContainerDied","Data":"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e"} Nov 03 17:55:41.846402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.846267 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" Nov 03 17:55:41.846402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.846291 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b" event={"ID":"eb99382e-9c30-4ee6-996f-2fdcb449f2ba","Type":"ContainerDied","Data":"e6ac4dbfda4c5b3e9f2d66ef23c18b5fdf169e7e46170de7f5ccc45104ff82f3"} Nov 03 17:55:41.846402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.846311 4885 scope.go:122] "RemoveContainer" containerID="0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e" Nov 03 17:55:41.854152 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.854135 4885 scope.go:122] "RemoveContainer" containerID="0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e" Nov 03 17:55:41.854405 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:41.854383 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e\": container with ID starting with 0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e not found: ID does not exist" containerID="0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e" Nov 03 17:55:41.854464 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:41.854416 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e"} err="failed to get container status \"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e\": rpc error: code = NotFound desc = could not find container \"0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e\": container with ID starting with 0d62d1e0833a96a04deb91c14d9f85af97fb988a6486bbe803a3a8a7e6b05e6e not found: ID does not exist" Nov 03 17:55:42.013467 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:42.013388 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:42.064510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:42.064475 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-909de6/noten-model-909de6-kserve-7465c4c76f-xpz2b"] Nov 03 17:55:43.660627 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:55:43.660596 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" path="/var/lib/kubelet/pods/eb99382e-9c30-4ee6-996f-2fdcb449f2ba/volumes" Nov 03 17:55:48.607439 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:55:48.607406 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:56:18.604697 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:56:18.604658 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:56:34.843115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:34.843062 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:56:38.766963 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:38.766879 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:56:38.767380 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:38.767111 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" containerID="cri-o://2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0" gracePeriod=30 Nov 03 17:56:44.021581 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.021550 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:56:44.075442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075367 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075442 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075425 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075458 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075495 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075538 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075571 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075617 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075607 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075779 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:56:44.075922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075798 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:56:44.075922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075812 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:56:44.075922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075853 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.075922 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075892 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\") pod \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\" (UID: \"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b\") " Nov 03 17:56:44.076215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.075941 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:56:44.076215 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.076178 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:56:44.076302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.076232 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.076302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.076252 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.076302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.076266 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.076302 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.076281 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.077989 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.077955 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:56:44.078137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.078033 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:56:44.078197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.078179 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:56:44.078265 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.078246 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv" pod "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" (UID: "3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b"). InnerVolumeSpecName "kube-api-access-hb2jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:56:44.107353 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.107325 4885 generic.go:356] "Generic (PLEG): container finished" podID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerID="2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0" exitCode=0 Nov 03 17:56:44.107498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.107399 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" Nov 03 17:56:44.107498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.107408 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerDied","Data":"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0"} Nov 03 17:56:44.107498 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.107478 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6" event={"ID":"3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b","Type":"ContainerDied","Data":"dd67b8f42d7c09e6b68b75844b943f596db2e1aa2aaca04c9a643f6a735d92e5"} Nov 03 17:56:44.107659 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.107506 4885 scope.go:122] "RemoveContainer" containerID="2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0" Nov 03 17:56:44.116046 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.116023 4885 scope.go:122] "RemoveContainer" containerID="c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce" Nov 03 17:56:44.123637 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.123619 4885 scope.go:122] "RemoveContainer" containerID="2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0" Nov 03 17:56:44.123908 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:56:44.123887 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0\": container with ID starting with 2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0 not found: ID does not exist" containerID="2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0" Nov 03 17:56:44.123953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.123920 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0"} err="failed to get container status \"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0\": rpc error: code = NotFound desc = could not find container \"2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0\": container with ID starting with 2b43a31a70594c8de98eba5afba35d95b79a8280e00ea68e423c57481a6ae1c0 not found: ID does not exist" Nov 03 17:56:44.123953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.123948 4885 scope.go:122] "RemoveContainer" containerID="c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce" Nov 03 17:56:44.124266 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:56:44.124244 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce\": container with ID starting with c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce not found: ID does not exist" containerID="c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce" Nov 03 17:56:44.124331 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.124275 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce"} err="failed to get container status \"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce\": rpc error: code = NotFound desc = could not find container \"c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce\": container with ID starting with c19da299657766e9baee8889fb641d6997545cb3247106f61caad88a471a2cce not found: ID does not exist" Nov 03 17:56:44.177504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.177479 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.177504 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.177502 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb2jv\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-kube-api-access-hb2jv\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.177658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.177513 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.177658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.177521 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.177658 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.177531 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:56:44.311551 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.311514 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:56:44.315200 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:44.315175 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-af9e3c-openshift-default-5fc5d986d5-5qlr6"] Nov 03 17:56:45.660950 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:45.660920 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" path="/var/lib/kubelet/pods/3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b/volumes" Nov 03 17:56:48.608858 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:56:48.608810 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:56:59.142664 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:56:59.142569 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:57:00.063420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:00.063385 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:57:00.182640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:00.182588 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" containerID="cri-o://dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af" gracePeriod=30 Nov 03 17:57:05.430955 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.430933 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:57:05.580882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.580797 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.580882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.580872 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.580909 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.580935 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.580984 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581007 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581034 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581086 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581136 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581129 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\") pod \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\" (UID: \"49f36c0e-3505-4dfb-8c4f-03e8eae7089e\") " Nov 03 17:57:05.581503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581342 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:57:05.581503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581373 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:05.581503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581435 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:57:05.581647 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581554 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:57:05.581840 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.581682 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:57:05.583401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.583381 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "kube-api-access-qgjsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:05.583624 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.583601 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:57:05.583829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.583803 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:57:05.583897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.583858 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token" pod "49f36c0e-3505-4dfb-8c4f-03e8eae7089e" (UID: "49f36c0e-3505-4dfb-8c4f-03e8eae7089e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:05.682739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682714 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682737 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682746 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682756 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682764 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682773 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682780 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682788 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjsh\" (UniqueName: \"kubernetes.io/projected/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-kube-api-access-qgjsh\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:05.682905 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:05.682798 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49f36c0e-3505-4dfb-8c4f-03e8eae7089e-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:06.208405 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.208366 4885 generic.go:356] "Generic (PLEG): container finished" podID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerID="dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af" exitCode=0 Nov 03 17:57:06.208670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.208423 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" Nov 03 17:57:06.208670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.208446 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" event={"ID":"49f36c0e-3505-4dfb-8c4f-03e8eae7089e","Type":"ContainerDied","Data":"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af"} Nov 03 17:57:06.208670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.208502 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28" event={"ID":"49f36c0e-3505-4dfb-8c4f-03e8eae7089e","Type":"ContainerDied","Data":"86de936dff38a7b2dec3058a1c50258465708a70753951e8f1e3be293470424f"} Nov 03 17:57:06.208670 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.208523 4885 scope.go:122] "RemoveContainer" containerID="dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af" Nov 03 17:57:06.216898 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.216878 4885 scope.go:122] "RemoveContainer" containerID="dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af" Nov 03 17:57:06.217203 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:57:06.217178 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af\": container with ID starting with dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af not found: ID does not exist" containerID="dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af" Nov 03 17:57:06.217305 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.217212 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af"} err="failed to get container status \"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af\": rpc error: code = NotFound desc = could not find container \"dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af\": container with ID starting with dcea52367ddccdb39faf6f8d8c276750f0a6589bf751174172f8ce341c5bc1af not found: ID does not exist" Nov 03 17:57:06.377460 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.377424 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:57:06.429610 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:06.429570 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-147834-openshift-default-5cf56689c9-b4q28"] Nov 03 17:57:07.660522 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:07.660484 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" path="/var/lib/kubelet/pods/49f36c0e-3505-4dfb-8c4f-03e8eae7089e/volumes" Nov 03 17:57:16.896127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896091 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896398 4885 cpu_manager.go:414] "containerMap: removing container" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896409 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896421 4885 cpu_manager.go:414] "containerMap: removing container" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896427 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896445 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896451 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896460 4885 cpu_manager.go:414] "containerMap: removing container" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896464 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896472 4885 cpu_manager.go:414] "containerMap: removing container" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896476 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896483 4885 cpu_manager.go:414] "containerMap: removing container" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896488 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896493 4885 cpu_manager.go:414] "containerMap: removing container" podUID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" containerName="main" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896498 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" containerName="main" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896503 4885 cpu_manager.go:414] "containerMap: removing container" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896508 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896514 4885 cpu_manager.go:414] "containerMap: removing container" podUID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896518 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896574 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896582 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd69a74-705a-4c8e-bf13-ff73aec9f58f" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896587 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f36c0e-3505-4dfb-8c4f-03e8eae7089e" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896593 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="98023514-7416-4f9b-a29d-cff2720447d3" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896598 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb99382e-9c30-4ee6-996f-2fdcb449f2ba" containerName="main" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896603 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea7afa7-5ff2-4cd7-8fbd-1f98b2e6049b" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896608 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce034f7-9564-43ee-a2f4-90f8ca8578b4" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896613 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="619c15b7-5793-4589-81b5-1a92f2c64741" containerName="istio-proxy" Nov 03 17:57:16.896646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.896618 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a06ce6-651d-461f-8631-c9dfedce328b" containerName="istio-proxy" Nov 03 17:57:16.899503 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.899482 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:16.974418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.974372 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:16.974583 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.974533 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhf5q\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:16.974583 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.974568 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:16.974677 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:16.974605 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.004774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004740 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-198bcf954428\"" Nov 03 17:57:17.004774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004740 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-dbae42e6384f\"" Nov 03 17:57:17.004774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004771 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"routing-overlay\"" Nov 03 17:57:17.005028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004809 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"agc-cred-bf3839ce3fb2\"" Nov 03 17:57:17.005028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004816 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"openshift-service-ca.crt\"" Nov 03 17:57:17.005028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004861 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-e2e--b603cb9e-dockercfg-78csf\"" Nov 03 17:57:17.005028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004894 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"kube-root-ca.crt\"" Nov 03 17:57:17.005028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.004902 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-8c9b0f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-8c9b0f\"" Nov 03 17:57:17.006758 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.006738 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:57:17.074985 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.074955 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhf5q\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.075142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.074990 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.075142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.075016 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.075142 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.075043 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.075854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.075794 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.075854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.075797 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.077807 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.077786 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.138164 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.138133 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhf5q\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.212103 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.212012 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:17.578624 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:57:17.578588 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6ba178_add1_4f46_a450_674d5ed237cf.slice/crio-d851c3ea95cb5b5776eb0e3036bfff64008bb08af4195cc2df2bbc2285b33e80 WatchSource:0}: Error finding container d851c3ea95cb5b5776eb0e3036bfff64008bb08af4195cc2df2bbc2285b33e80: Status 404 returned error can't find the container with id d851c3ea95cb5b5776eb0e3036bfff64008bb08af4195cc2df2bbc2285b33e80 Nov 03 17:57:17.580273 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:17.580254 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:57:18.259904 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:18.259871 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" event={"ID":"ac6ba178-add1-4f46-a450-674d5ed237cf","Type":"ContainerStarted","Data":"51b93cb492bd6bf3fe6a97561854f81837e7abc4b23fe5a7c84866a55cba4498"} Nov 03 17:57:18.260317 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:18.259918 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" event={"ID":"ac6ba178-add1-4f46-a450-674d5ed237cf","Type":"ContainerStarted","Data":"d851c3ea95cb5b5776eb0e3036bfff64008bb08af4195cc2df2bbc2285b33e80"} Nov 03 17:57:18.260317 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:18.259990 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:57:18.382967 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:18.382923 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podStartSLOduration=2.382909377 podStartE2EDuration="2.382909377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:18.381721512 +0000 UTC m=+2380.836410538" watchObservedRunningTime="2026-11-03 17:57:18.382909377 +0000 UTC m=+2380.837598401" Nov 03 17:57:18.604596 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:57:18.604502 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:57:19.450366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:19.450327 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:57:22.221134 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.221094 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz"] Nov 03 17:57:22.224899 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.224880 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.276810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.276781 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz"] Nov 03 17:57:22.278910 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.278880 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:22.318015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.317979 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.318015 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.318017 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqbm\" (UniqueName: \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.318247 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.318045 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.326759 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.326716 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:57:22.326901 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.326838 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:22.326901 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.326847 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:22.419577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.419531 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.419577 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.419567 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqbm\" (UniqueName: \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.419776 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.419592 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.420287 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.420223 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.422115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.422092 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.482517 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.482449 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqbm\" (UniqueName: \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\") pod \"maas-api-revoke-sub-5e87bf8a-twfxz\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.537625 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.537597 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:22.900386 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:57:22.900349 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3383005_59b2_45c3_abf9_21e5b7c528b6.slice/crio-d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921 WatchSource:0}: Error finding container d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921: Status 404 returned error can't find the container with id d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921 Nov 03 17:57:22.902881 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:22.902854 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz"] Nov 03 17:57:23.285815 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:23.285783 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" event={"ID":"c3383005-59b2-45c3-abf9-21e5b7c528b6","Type":"ContainerStarted","Data":"b324b5c52c39c09e41b7e5dc0efceb83f5e176cc2d861deb15e59077e3204d1f"} Nov 03 17:57:23.286339 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:23.285830 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" event={"ID":"c3383005-59b2-45c3-abf9-21e5b7c528b6","Type":"ContainerStarted","Data":"d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921"} Nov 03 17:57:23.410185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:23.410132 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" podStartSLOduration=1.410117651 podStartE2EDuration="1.410117651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:23.409047601 +0000 UTC m=+2385.863736637" watchObservedRunningTime="2026-11-03 17:57:23.410117651 +0000 UTC m=+2385.864806714" Nov 03 17:57:24.291028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:24.290992 4885 generic.go:356] "Generic (PLEG): container finished" podID="c3383005-59b2-45c3-abf9-21e5b7c528b6" containerID="b324b5c52c39c09e41b7e5dc0efceb83f5e176cc2d861deb15e59077e3204d1f" exitCode=0 Nov 03 17:57:24.291028 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:24.291030 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" event={"ID":"c3383005-59b2-45c3-abf9-21e5b7c528b6","Type":"ContainerDied","Data":"b324b5c52c39c09e41b7e5dc0efceb83f5e176cc2d861deb15e59077e3204d1f"} Nov 03 17:57:25.421689 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.421656 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:25.447725 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.447696 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\") pod \"c3383005-59b2-45c3-abf9-21e5b7c528b6\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " Nov 03 17:57:25.447854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.447748 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\") pod \"c3383005-59b2-45c3-abf9-21e5b7c528b6\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " Nov 03 17:57:25.447854 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.447801 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\" (UniqueName: \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\") pod \"c3383005-59b2-45c3-abf9-21e5b7c528b6\" (UID: \"c3383005-59b2-45c3-abf9-21e5b7c528b6\") " Nov 03 17:57:25.448127 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.448102 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca" pod "c3383005-59b2-45c3-abf9-21e5b7c528b6" (UID: "c3383005-59b2-45c3-abf9-21e5b7c528b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:25.449940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.449912 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca" pod "c3383005-59b2-45c3-abf9-21e5b7c528b6" (UID: "c3383005-59b2-45c3-abf9-21e5b7c528b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:25.450044 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.449958 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm" pod "c3383005-59b2-45c3-abf9-21e5b7c528b6" (UID: "c3383005-59b2-45c3-abf9-21e5b7c528b6"). InnerVolumeSpecName "kube-api-access-tvqbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:25.548382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.548302 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:25.548382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.548331 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3383005-59b2-45c3-abf9-21e5b7c528b6-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:25.548382 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:25.548343 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqbm\" (UniqueName: \"kubernetes.io/projected/c3383005-59b2-45c3-abf9-21e5b7c528b6-kube-api-access-tvqbm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:26.301140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:26.301096 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" event={"ID":"c3383005-59b2-45c3-abf9-21e5b7c528b6","Type":"ContainerDied","Data":"d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921"} Nov 03 17:57:26.301290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:26.301160 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d95cdf98ecc86fd52ef91cac5722bb0f46bbfa887d2d9fd41a7730251b2921" Nov 03 17:57:26.301290 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:26.301118 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz" Nov 03 17:57:29.267597 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:29.267564 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:57:34.691083 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.691035 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg"] Nov 03 17:57:34.691470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.691362 4885 cpu_manager.go:414] "containerMap: removing container" podUID="c3383005-59b2-45c3-abf9-21e5b7c528b6" containerName="revoke-keys" Nov 03 17:57:34.691470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.691372 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3383005-59b2-45c3-abf9-21e5b7c528b6" containerName="revoke-keys" Nov 03 17:57:34.691470 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.691448 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3383005-59b2-45c3-abf9-21e5b7c528b6" containerName="revoke-keys" Nov 03 17:57:34.694274 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.694258 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.725000 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.724970 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.725141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.725049 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcs4\" (UniqueName: \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.725141 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.725121 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.748644 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.748619 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg"] Nov 03 17:57:34.798535 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.798496 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:34.798752 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.798565 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:57:34.798752 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.798596 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:34.798752 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.798656 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:34.826186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.826152 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcs4\" (UniqueName: \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.826186 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.826188 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.826449 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.826232 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.826821 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.826802 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.828817 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.828793 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:34.888918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:34.888890 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcs4\" (UniqueName: \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\") pod \"maas-api-revoke-sub-f67ac462-v56rg\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:35.006695 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:35.006624 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:35.372619 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:57:35.372589 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d9df1_cb35_4474_b6c9_abd29efea935.slice/crio-3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137 WatchSource:0}: Error finding container 3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137: Status 404 returned error can't find the container with id 3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137 Nov 03 17:57:35.374995 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:35.374973 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg"] Nov 03 17:57:36.343608 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:36.343569 4885 generic.go:356] "Generic (PLEG): container finished" podID="9d8d9df1-cb35-4474-b6c9-abd29efea935" containerID="85c3843e8058a9dc594478de2ec2802d2dcb6fed75093fa2bb22e17f47b62856" exitCode=0 Nov 03 17:57:36.343979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:36.343626 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" event={"ID":"9d8d9df1-cb35-4474-b6c9-abd29efea935","Type":"ContainerDied","Data":"85c3843e8058a9dc594478de2ec2802d2dcb6fed75093fa2bb22e17f47b62856"} Nov 03 17:57:36.343979 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:36.343668 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" event={"ID":"9d8d9df1-cb35-4474-b6c9-abd29efea935","Type":"ContainerStarted","Data":"3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137"} Nov 03 17:57:37.475128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.475103 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:37.549410 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.549383 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\") pod \"9d8d9df1-cb35-4474-b6c9-abd29efea935\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " Nov 03 17:57:37.549540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.549477 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\" (UniqueName: \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\") pod \"9d8d9df1-cb35-4474-b6c9-abd29efea935\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " Nov 03 17:57:37.549540 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.549507 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\") pod \"9d8d9df1-cb35-4474-b6c9-abd29efea935\" (UID: \"9d8d9df1-cb35-4474-b6c9-abd29efea935\") " Nov 03 17:57:37.549706 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.549687 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca" pod "9d8d9df1-cb35-4474-b6c9-abd29efea935" (UID: "9d8d9df1-cb35-4474-b6c9-abd29efea935"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:37.551584 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.551559 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4" pod "9d8d9df1-cb35-4474-b6c9-abd29efea935" (UID: "9d8d9df1-cb35-4474-b6c9-abd29efea935"). InnerVolumeSpecName "kube-api-access-tmcs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:37.551761 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.551741 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca" pod "9d8d9df1-cb35-4474-b6c9-abd29efea935" (UID: "9d8d9df1-cb35-4474-b6c9-abd29efea935"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:37.650052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.650024 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcs4\" (UniqueName: \"kubernetes.io/projected/9d8d9df1-cb35-4474-b6c9-abd29efea935-kube-api-access-tmcs4\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:37.650052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.650049 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:37.650052 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:37.650058 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d9df1-cb35-4474-b6c9-abd29efea935-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:38.352953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:38.352919 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" Nov 03 17:57:38.352953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:38.352929 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg" event={"ID":"9d8d9df1-cb35-4474-b6c9-abd29efea935","Type":"ContainerDied","Data":"3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137"} Nov 03 17:57:38.353246 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:38.352977 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6bb5f1111d60a4ba2b76a6d6cbd1e9b411bce47ecb1c74bac9f502b2f8d137" Nov 03 17:57:39.265108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:39.265058 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:57:46.529238 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.529201 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8"] Nov 03 17:57:46.529642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.529531 4885 cpu_manager.go:414] "containerMap: removing container" podUID="9d8d9df1-cb35-4474-b6c9-abd29efea935" containerName="revoke-keys" Nov 03 17:57:46.529642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.529543 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8d9df1-cb35-4474-b6c9-abd29efea935" containerName="revoke-keys" Nov 03 17:57:46.529642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.529593 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8d9df1-cb35-4474-b6c9-abd29efea935" containerName="revoke-keys" Nov 03 17:57:46.532472 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.532457 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.586771 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.586740 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8"] Nov 03 17:57:46.624934 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.624903 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.625146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.624950 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.625146 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.624974 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhgm\" (UniqueName: \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.636538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.636508 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:46.636538 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.636524 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:46.636705 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.636605 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:46.637321 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.637307 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:57:46.726228 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.726189 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.726418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.726248 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.726418 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.726271 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhgm\" (UniqueName: \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.726874 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.726845 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.728773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.728755 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.790296 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.790221 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhgm\" (UniqueName: \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\") pod \"maas-api-revoke-sub-ac7c7814-wn6g8\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:46.844903 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:46.844874 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:47.217386 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:57:47.217357 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fce8d73_3572_4451_8a32_30f0f7e57662.slice/crio-38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd WatchSource:0}: Error finding container 38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd: Status 404 returned error can't find the container with id 38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd Nov 03 17:57:47.218782 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:47.218756 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8"] Nov 03 17:57:47.389291 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:47.389257 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" event={"ID":"7fce8d73-3572-4451-8a32-30f0f7e57662","Type":"ContainerStarted","Data":"38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd"} Nov 03 17:57:48.395262 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:48.395226 4885 generic.go:356] "Generic (PLEG): container finished" podID="7fce8d73-3572-4451-8a32-30f0f7e57662" containerID="84f1c19c49d207670d6eebd2a4b24daf3101c783ea1866d6ad39a3bc27c79fb2" exitCode=0 Nov 03 17:57:48.395638 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:48.395285 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" event={"ID":"7fce8d73-3572-4451-8a32-30f0f7e57662","Type":"ContainerDied","Data":"84f1c19c49d207670d6eebd2a4b24daf3101c783ea1866d6ad39a3bc27c79fb2"} Nov 03 17:57:48.604246 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:57:48.604213 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:57:49.265758 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.265712 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:57:49.527563 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.527507 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:49.651212 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.651172 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\") pod \"7fce8d73-3572-4451-8a32-30f0f7e57662\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " Nov 03 17:57:49.651401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.651231 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\" (UniqueName: \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\") pod \"7fce8d73-3572-4451-8a32-30f0f7e57662\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " Nov 03 17:57:49.651401 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.651294 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\") pod \"7fce8d73-3572-4451-8a32-30f0f7e57662\" (UID: \"7fce8d73-3572-4451-8a32-30f0f7e57662\") " Nov 03 17:57:49.651747 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.651718 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca" pod "7fce8d73-3572-4451-8a32-30f0f7e57662" (UID: "7fce8d73-3572-4451-8a32-30f0f7e57662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:49.653374 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.653354 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca" pod "7fce8d73-3572-4451-8a32-30f0f7e57662" (UID: "7fce8d73-3572-4451-8a32-30f0f7e57662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:49.653510 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.653488 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm" pod "7fce8d73-3572-4451-8a32-30f0f7e57662" (UID: "7fce8d73-3572-4451-8a32-30f0f7e57662"). InnerVolumeSpecName "kube-api-access-hxhgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:49.752750 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.752719 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhgm\" (UniqueName: \"kubernetes.io/projected/7fce8d73-3572-4451-8a32-30f0f7e57662-kube-api-access-hxhgm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:49.752750 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.752743 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:49.752750 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:49.752753 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fce8d73-3572-4451-8a32-30f0f7e57662-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:57:50.404885 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:50.404856 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" Nov 03 17:57:50.405056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:50.404860 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8" event={"ID":"7fce8d73-3572-4451-8a32-30f0f7e57662","Type":"ContainerDied","Data":"38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd"} Nov 03 17:57:50.405056 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:50.404981 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ec8425ea3ce43f5f9a9e1722b722a9af1ffd477fb9e04bef7f777fa09778dd" Nov 03 17:57:59.265566 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:57:59.265526 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:02.011380 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.011345 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95"] Nov 03 17:58:02.011763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.011671 4885 cpu_manager.go:414] "containerMap: removing container" podUID="7fce8d73-3572-4451-8a32-30f0f7e57662" containerName="revoke-keys" Nov 03 17:58:02.011763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.011683 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fce8d73-3572-4451-8a32-30f0f7e57662" containerName="revoke-keys" Nov 03 17:58:02.011763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.011726 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fce8d73-3572-4451-8a32-30f0f7e57662" containerName="revoke-keys" Nov 03 17:58:02.014857 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.014842 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.069477 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.069449 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95"] Nov 03 17:58:02.118841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.118810 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:02.118841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.118818 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:02.118841 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.118845 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:02.120452 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.120437 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:58:02.160851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.160810 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.161014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.160869 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjclj\" (UniqueName: \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.161014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.160999 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.262474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.262385 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.262474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.262431 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjclj\" (UniqueName: \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.262474 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.262468 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.263204 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.263172 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.264882 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.264863 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.325482 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.325447 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjclj\" (UniqueName: \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\") pod \"maas-api-revoke-sub-23bc1f8d-xxd95\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.327140 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.327109 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:02.698732 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:58:02.698700 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6021d689_c344_4996_81b5_46cfc4d2cec1.slice/crio-87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b WatchSource:0}: Error finding container 87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b: Status 404 returned error can't find the container with id 87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b Nov 03 17:58:02.701001 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.700978 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95"] Nov 03 17:58:02.862561 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:02.862529 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:58:03.465408 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:03.465369 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" event={"ID":"6021d689-c344-4996-81b5-46cfc4d2cec1","Type":"ContainerStarted","Data":"32b415052e63b0348c93186becbaed2e58f8bb69a71039ec696f1872e57a76d0"} Nov 03 17:58:03.465895 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:03.465433 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" event={"ID":"6021d689-c344-4996-81b5-46cfc4d2cec1","Type":"ContainerStarted","Data":"87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b"} Nov 03 17:58:03.584794 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:03.584744 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" podStartSLOduration=2.584726857 podStartE2EDuration="2.584726857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:03.584105292 +0000 UTC m=+2426.038794313" watchObservedRunningTime="2026-11-03 17:58:03.584726857 +0000 UTC m=+2426.039415882" Nov 03 17:58:04.471533 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:04.471500 4885 generic.go:356] "Generic (PLEG): container finished" podID="6021d689-c344-4996-81b5-46cfc4d2cec1" containerID="32b415052e63b0348c93186becbaed2e58f8bb69a71039ec696f1872e57a76d0" exitCode=0 Nov 03 17:58:04.471918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:04.471578 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" event={"ID":"6021d689-c344-4996-81b5-46cfc4d2cec1","Type":"ContainerDied","Data":"32b415052e63b0348c93186becbaed2e58f8bb69a71039ec696f1872e57a76d0"} Nov 03 17:58:05.603774 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.603749 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:05.691952 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.691921 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\") pod \"6021d689-c344-4996-81b5-46cfc4d2cec1\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " Nov 03 17:58:05.692128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.691997 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\") pod \"6021d689-c344-4996-81b5-46cfc4d2cec1\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " Nov 03 17:58:05.692128 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.692027 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\" (UniqueName: \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\") pod \"6021d689-c344-4996-81b5-46cfc4d2cec1\" (UID: \"6021d689-c344-4996-81b5-46cfc4d2cec1\") " Nov 03 17:58:05.692375 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.692346 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca" pod "6021d689-c344-4996-81b5-46cfc4d2cec1" (UID: "6021d689-c344-4996-81b5-46cfc4d2cec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:05.694168 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.694140 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca" pod "6021d689-c344-4996-81b5-46cfc4d2cec1" (UID: "6021d689-c344-4996-81b5-46cfc4d2cec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:58:05.694253 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.694206 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj" pod "6021d689-c344-4996-81b5-46cfc4d2cec1" (UID: "6021d689-c344-4996-81b5-46cfc4d2cec1"). InnerVolumeSpecName "kube-api-access-qjclj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:05.792683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.792615 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:05.792683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.792644 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjclj\" (UniqueName: \"kubernetes.io/projected/6021d689-c344-4996-81b5-46cfc4d2cec1-kube-api-access-qjclj\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:05.792683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:05.792659 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6021d689-c344-4996-81b5-46cfc4d2cec1-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:06.483484 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:06.483456 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" Nov 03 17:58:06.483663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:06.483453 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95" event={"ID":"6021d689-c344-4996-81b5-46cfc4d2cec1","Type":"ContainerDied","Data":"87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b"} Nov 03 17:58:06.483663 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:06.483575 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87c4b1a21955f3dbb4aeab8c6065df1caf18b122e64a371cdf9df1a7fcff2f8b" Nov 03 17:58:09.266014 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:09.265978 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:15.064107 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.064060 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn"] Nov 03 17:58:15.064489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.064396 4885 cpu_manager.go:414] "containerMap: removing container" podUID="6021d689-c344-4996-81b5-46cfc4d2cec1" containerName="revoke-keys" Nov 03 17:58:15.064489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.064406 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6021d689-c344-4996-81b5-46cfc4d2cec1" containerName="revoke-keys" Nov 03 17:58:15.064489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.064459 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="6021d689-c344-4996-81b5-46cfc4d2cec1" containerName="revoke-keys" Nov 03 17:58:15.067372 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.067356 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.075396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.075379 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.075486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.075432 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqgf6\" (UniqueName: \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.075486 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.075460 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.121402 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.121372 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn"] Nov 03 17:58:15.170978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.170943 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:15.171137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.170995 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:15.171137 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.171042 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:58:15.172475 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.172462 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:15.175851 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.175829 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqgf6\" (UniqueName: \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.175947 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.175866 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.175947 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.175899 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.176466 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.176438 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.178665 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.178646 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.264251 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.264221 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqgf6\" (UniqueName: \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\") pod \"maas-api-revoke-sub-ec87d6fd-2s8wn\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.379443 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.379413 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:15.379936 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.379911 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:58:15.381284 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.381252 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:15.415094 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.415050 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:15.418677 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.418655 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.474765 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.474700 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:15.477544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.477460 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.477544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.477505 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvxh\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.477544 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.477526 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.477739 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.477646 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.577994 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.577958 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.578184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.578029 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.578184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.578060 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvxh\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.578184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.578111 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.578763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.578742 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.578810 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.578750 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.580650 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.580631 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.641576 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.641499 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvxh\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.733679 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.733650 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:15.766927 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:15.766891 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn"] Nov 03 17:58:15.767288 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:58:15.767263 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c4c8d4_c6e1_4896_b6dc_8792960c7b35.slice/crio-a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b WatchSource:0}: Error finding container a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b: Status 404 returned error can't find the container with id a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b Nov 03 17:58:16.102604 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:58:16.102575 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ca5069_7166_4708_b3b0_9fd075788aec.slice/crio-33ec1fd69d2486142460363cf14fc565f140ffbcdfc134a3d871b8b23d22daec WatchSource:0}: Error finding container 33ec1fd69d2486142460363cf14fc565f140ffbcdfc134a3d871b8b23d22daec: Status 404 returned error can't find the container with id 33ec1fd69d2486142460363cf14fc565f140ffbcdfc134a3d871b8b23d22daec Nov 03 17:58:16.104812 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.104786 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:16.526558 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.526516 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" event={"ID":"45ca5069-7166-4708-b3b0-9fd075788aec","Type":"ContainerStarted","Data":"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99"} Nov 03 17:58:16.526755 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.526576 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" event={"ID":"45ca5069-7166-4708-b3b0-9fd075788aec","Type":"ContainerStarted","Data":"33ec1fd69d2486142460363cf14fc565f140ffbcdfc134a3d871b8b23d22daec"} Nov 03 17:58:16.526755 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.526694 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:16.528195 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.528169 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" event={"ID":"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35","Type":"ContainerStarted","Data":"5589de9dd04be6c3efd771b573ea6b0d519e344bc3bcd316bda513f7ea0a2673"} Nov 03 17:58:16.528318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.528214 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" event={"ID":"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35","Type":"ContainerStarted","Data":"a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b"} Nov 03 17:58:16.647424 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.647371 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" podStartSLOduration=1.647355262 podStartE2EDuration="1.647355262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:16.646365301 +0000 UTC m=+2439.101054326" watchObservedRunningTime="2026-11-03 17:58:16.647355262 +0000 UTC m=+2439.102044278" Nov 03 17:58:16.766259 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.766203 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" podStartSLOduration=1.7661888860000001 podStartE2EDuration="1.766188886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:16.764654422 +0000 UTC m=+2439.219343444" watchObservedRunningTime="2026-11-03 17:58:16.766188886 +0000 UTC m=+2439.220877908" Nov 03 17:58:16.968495 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:16.968458 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:58:17.533756 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:17.533726 4885 generic.go:356] "Generic (PLEG): container finished" podID="a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" containerID="5589de9dd04be6c3efd771b573ea6b0d519e344bc3bcd316bda513f7ea0a2673" exitCode=0 Nov 03 17:58:17.534176 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:17.533778 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" event={"ID":"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35","Type":"ContainerDied","Data":"5589de9dd04be6c3efd771b573ea6b0d519e344bc3bcd316bda513f7ea0a2673"} Nov 03 17:58:18.610369 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:58:18.610322 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:58:18.662398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.662378 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:18.801715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.801633 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\") pod \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " Nov 03 17:58:18.801715 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.801683 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\") pod \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " Nov 03 17:58:18.801948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.801865 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\" (UniqueName: \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\") pod \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\" (UID: \"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35\") " Nov 03 17:58:18.802018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.801998 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca" pod "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" (UID: "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:18.802161 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.802143 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:18.803862 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.803841 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca" pod "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" (UID: "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:58:18.804013 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.803995 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6" pod "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" (UID: "a5c4c8d4-c6e1-4896-b6dc-8792960c7b35"). InnerVolumeSpecName "kube-api-access-hqgf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:18.902609 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.902575 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:18.902609 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:18.902603 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqgf6\" (UniqueName: \"kubernetes.io/projected/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35-kube-api-access-hqgf6\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:19.543865 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:19.543829 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" Nov 03 17:58:19.544085 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:19.543864 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn" event={"ID":"a5c4c8d4-c6e1-4896-b6dc-8792960c7b35","Type":"ContainerDied","Data":"a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b"} Nov 03 17:58:19.544085 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:19.543910 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a7f22854b7333a17a09c017ba55e150446f52bc4703a86ed568bf247adaa1b" Nov 03 17:58:20.380627 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.380582 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" containerID="cri-o://51b93cb492bd6bf3fe6a97561854f81837e7abc4b23fe5a7c84866a55cba4498" gracePeriod=25 Nov 03 17:58:20.552237 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.552205 4885 generic.go:356] "Generic (PLEG): container finished" podID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerID="51b93cb492bd6bf3fe6a97561854f81837e7abc4b23fe5a7c84866a55cba4498" exitCode=0 Nov 03 17:58:20.552366 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.552267 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" event={"ID":"ac6ba178-add1-4f46-a450-674d5ed237cf","Type":"ContainerDied","Data":"51b93cb492bd6bf3fe6a97561854f81837e7abc4b23fe5a7c84866a55cba4498"} Nov 03 17:58:20.622499 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.622478 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:58:20.713773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.713692 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\") pod \"ac6ba178-add1-4f46-a450-674d5ed237cf\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " Nov 03 17:58:20.713773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.713733 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\") pod \"ac6ba178-add1-4f46-a450-674d5ed237cf\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " Nov 03 17:58:20.713773 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.713755 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\") pod \"ac6ba178-add1-4f46-a450-674d5ed237cf\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " Nov 03 17:58:20.713989 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.713781 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\") pod \"ac6ba178-add1-4f46-a450-674d5ed237cf\" (UID: \"ac6ba178-add1-4f46-a450-674d5ed237cf\") " Nov 03 17:58:20.714158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.714135 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume" pod "ac6ba178-add1-4f46-a450-674d5ed237cf" (UID: "ac6ba178-add1-4f46-a450-674d5ed237cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:20.714236 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.714172 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume" pod "ac6ba178-add1-4f46-a450-674d5ed237cf" (UID: "ac6ba178-add1-4f46-a450-674d5ed237cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:20.716004 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.715976 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials" pod "ac6ba178-add1-4f46-a450-674d5ed237cf" (UID: "ac6ba178-add1-4f46-a450-674d5ed237cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:20.716145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.716038 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q" pod "ac6ba178-add1-4f46-a450-674d5ed237cf" (UID: "ac6ba178-add1-4f46-a450-674d5ed237cf"). InnerVolumeSpecName "kube-api-access-rhf5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:20.815010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.814976 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:20.815010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.815004 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:20.815010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.815013 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhf5q\" (UniqueName: \"kubernetes.io/projected/ac6ba178-add1-4f46-a450-674d5ed237cf-kube-api-access-rhf5q\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:20.815010 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:20.815022 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ac6ba178-add1-4f46-a450-674d5ed237cf-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:21.557834 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:21.557804 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" Nov 03 17:58:21.557834 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:21.557818 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr" event={"ID":"ac6ba178-add1-4f46-a450-674d5ed237cf","Type":"ContainerDied","Data":"d851c3ea95cb5b5776eb0e3036bfff64008bb08af4195cc2df2bbc2285b33e80"} Nov 03 17:58:21.558307 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:21.557884 4885 scope.go:122] "RemoveContainer" containerID="51b93cb492bd6bf3fe6a97561854f81837e7abc4b23fe5a7c84866a55cba4498" Nov 03 17:58:21.726850 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:21.726816 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:58:21.778224 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:21.778192 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-fc8frnzjr"] Nov 03 17:58:23.660908 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:23.660875 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" path="/var/lib/kubelet/pods/ac6ba178-add1-4f46-a450-674d5ed237cf/volumes" Nov 03 17:58:27.536291 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:27.536258 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:28.116131 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116100 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq"] Nov 03 17:58:28.116431 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116420 4885 cpu_manager.go:414] "containerMap: removing container" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" Nov 03 17:58:28.116490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116434 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" Nov 03 17:58:28.116490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116448 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" containerName="revoke-keys" Nov 03 17:58:28.116490 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116453 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" containerName="revoke-keys" Nov 03 17:58:28.116594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116517 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" containerName="revoke-keys" Nov 03 17:58:28.116594 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.116529 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6ba178-add1-4f46-a450-674d5ed237cf" containerName="payload-processing" Nov 03 17:58:28.120986 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.120968 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.166247 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.166212 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.166412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.166254 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.166412 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.166313 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-548c9\" (UniqueName: \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.174398 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.174372 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq"] Nov 03 17:58:28.224185 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.224154 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:28.224351 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.224228 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:28.225849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.225835 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:28.225913 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.225892 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:58:28.267414 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.267380 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.267554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.267422 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.267554 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.267462 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-548c9\" (UniqueName: \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.268115 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.268097 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.269993 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.269972 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.330018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.329983 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-548c9\" (UniqueName: \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\") pod \"maas-api-revoke-sub-d6b1784a-ghcqq\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.353951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.353924 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:28.355362 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.355330 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:28.389710 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.389680 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:28.393030 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.393015 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.433528 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.433504 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:28.450337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.450307 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:28.470354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.470321 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.470537 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.470386 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.470537 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.470458 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.470696 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.470551 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwq6n\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571198 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571163 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwq6n\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571553 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571239 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571553 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571266 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571553 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571282 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571832 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.571965 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.571946 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.573762 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.573744 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.633649 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.633611 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwq6n\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\") pod \"payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.705576 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.705504 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:28.807933 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:58:28.807897 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078ef6e5_eb34_4c63_95cd_b37eb00b7b9f.slice/crio-4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c WatchSource:0}: Error finding container 4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c: Status 404 returned error can't find the container with id 4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c Nov 03 17:58:28.809669 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:28.809649 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq"] Nov 03 17:58:29.085893 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.085867 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:29.593588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.593555 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" event={"ID":"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f","Type":"ContainerStarted","Data":"29f96ab9d2df944372f234bd16a0274566c134af626d4203f9e50b2138184112"} Nov 03 17:58:29.594101 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.593612 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" event={"ID":"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f","Type":"ContainerStarted","Data":"4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c"} Nov 03 17:58:29.595037 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.595015 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" event={"ID":"2d343aa5-f3de-4b4f-adec-13c9d9bd4828","Type":"ContainerStarted","Data":"17cb86137ff7430f5b55ff409a05190c1124d0b14efd3f6996bee0d25205a006"} Nov 03 17:58:29.595175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.595058 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" event={"ID":"2d343aa5-f3de-4b4f-adec-13c9d9bd4828","Type":"ContainerStarted","Data":"61450a7bf684be57db2d9b538c6fa5fd4742da501b1659dc19fc10a73c8a97e4"} Nov 03 17:58:29.595175 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.595126 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:29.713970 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.713922 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" podStartSLOduration=1.713910282 podStartE2EDuration="1.713910282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:29.712206686 +0000 UTC m=+2452.166895712" watchObservedRunningTime="2026-11-03 17:58:29.713910282 +0000 UTC m=+2452.168599305" Nov 03 17:58:29.832324 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:29.832264 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" podStartSLOduration=1.832249062 podStartE2EDuration="1.832249062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:29.83196236 +0000 UTC m=+2452.286651391" watchObservedRunningTime="2026-11-03 17:58:29.832249062 +0000 UTC m=+2452.286938089" Nov 03 17:58:30.258852 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:30.258818 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:58:30.602276 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:30.602187 4885 generic.go:356] "Generic (PLEG): container finished" podID="078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" containerID="29f96ab9d2df944372f234bd16a0274566c134af626d4203f9e50b2138184112" exitCode=0 Nov 03 17:58:30.602637 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:30.602271 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" event={"ID":"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f","Type":"ContainerDied","Data":"29f96ab9d2df944372f234bd16a0274566c134af626d4203f9e50b2138184112"} Nov 03 17:58:31.736429 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.736409 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:31.805718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.805684 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\") pod \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " Nov 03 17:58:31.805718 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.805724 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\") pod \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " Nov 03 17:58:31.805969 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.805776 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\" (UniqueName: \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\") pod \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\" (UID: \"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f\") " Nov 03 17:58:31.806165 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.806130 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca" pod "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" (UID: "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:31.807935 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.807906 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca" pod "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" (UID: "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:58:31.808039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.807958 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9" pod "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" (UID: "078ef6e5-eb34-4c63-95cd-b37eb00b7b9f"). InnerVolumeSpecName "kube-api-access-548c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:31.907005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.906967 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:31.907005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.906996 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:31.907005 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:31.907005 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-548c9\" (UniqueName: \"kubernetes.io/projected/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f-kube-api-access-548c9\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:32.612825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:32.612794 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" event={"ID":"078ef6e5-eb34-4c63-95cd-b37eb00b7b9f","Type":"ContainerDied","Data":"4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c"} Nov 03 17:58:32.612825 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:32.612810 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq" Nov 03 17:58:32.613039 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:32.612846 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4352d249423579dbd3ac8c147fd80e86ed10a32738c39b55dd779273d0c3b76c" Nov 03 17:58:33.355184 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.355143 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" containerID="cri-o://ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99" gracePeriod=25 Nov 03 17:58:33.604465 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.604439 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:33.618945 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.618914 4885 generic.go:356] "Generic (PLEG): container finished" podID="45ca5069-7166-4708-b3b0-9fd075788aec" containerID="ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99" exitCode=0 Nov 03 17:58:33.619098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.618962 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" Nov 03 17:58:33.619098 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.618966 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" event={"ID":"45ca5069-7166-4708-b3b0-9fd075788aec","Type":"ContainerDied","Data":"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99"} Nov 03 17:58:33.619230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.619131 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs" event={"ID":"45ca5069-7166-4708-b3b0-9fd075788aec","Type":"ContainerDied","Data":"33ec1fd69d2486142460363cf14fc565f140ffbcdfc134a3d871b8b23d22daec"} Nov 03 17:58:33.619230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.619168 4885 scope.go:122] "RemoveContainer" containerID="ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99" Nov 03 17:58:33.627631 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.627609 4885 scope.go:122] "RemoveContainer" containerID="ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99" Nov 03 17:58:33.627935 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:58:33.627917 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99\": container with ID starting with ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99 not found: ID does not exist" containerID="ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99" Nov 03 17:58:33.627990 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.627945 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99"} err="failed to get container status \"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99\": rpc error: code = NotFound desc = could not find container \"ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99\": container with ID starting with ceabc61ffd3e883469077f30cabd9b4a516dfcbf58d3d296db3504ed0a5a0d99 not found: ID does not exist" Nov 03 17:58:33.721951 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.721921 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\") pod \"45ca5069-7166-4708-b3b0-9fd075788aec\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " Nov 03 17:58:33.722145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.721984 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\") pod \"45ca5069-7166-4708-b3b0-9fd075788aec\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " Nov 03 17:58:33.722145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.722021 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\") pod \"45ca5069-7166-4708-b3b0-9fd075788aec\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " Nov 03 17:58:33.722145 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.722086 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\") pod \"45ca5069-7166-4708-b3b0-9fd075788aec\" (UID: \"45ca5069-7166-4708-b3b0-9fd075788aec\") " Nov 03 17:58:33.722364 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.722335 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume" pod "45ca5069-7166-4708-b3b0-9fd075788aec" (UID: "45ca5069-7166-4708-b3b0-9fd075788aec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:33.722489 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.722443 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume" pod "45ca5069-7166-4708-b3b0-9fd075788aec" (UID: "45ca5069-7166-4708-b3b0-9fd075788aec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:33.725640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.725610 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh" pod "45ca5069-7166-4708-b3b0-9fd075788aec" (UID: "45ca5069-7166-4708-b3b0-9fd075788aec"). InnerVolumeSpecName "kube-api-access-hdvxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:33.726108 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.726089 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials" pod "45ca5069-7166-4708-b3b0-9fd075788aec" (UID: "45ca5069-7166-4708-b3b0-9fd075788aec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:33.823151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.823118 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:33.823151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.823145 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:33.823151 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.823155 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvxh\" (UniqueName: \"kubernetes.io/projected/45ca5069-7166-4708-b3b0-9fd075788aec-kube-api-access-hdvxh\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:33.823390 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:33.823163 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45ca5069-7166-4708-b3b0-9fd075788aec-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:34.042791 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:34.042749 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:34.048034 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:34.048011 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-75bb4zhbs"] Nov 03 17:58:35.660437 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:35.660405 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" path="/var/lib/kubelet/pods/45ca5069-7166-4708-b3b0-9fd075788aec/volumes" Nov 03 17:58:40.450354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:40.450324 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:40.453029 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:40.453000 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:58:43.078948 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:43.078909 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:58:45.451588 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.451539 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerName="payload-processing" containerID="cri-o://17cb86137ff7430f5b55ff409a05190c1124d0b14efd3f6996bee0d25205a006" gracePeriod=25 Nov 03 17:58:45.673829 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.673802 4885 generic.go:356] "Generic (PLEG): container finished" podID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerID="17cb86137ff7430f5b55ff409a05190c1124d0b14efd3f6996bee0d25205a006" exitCode=0 Nov 03 17:58:45.673972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.673856 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" event={"ID":"2d343aa5-f3de-4b4f-adec-13c9d9bd4828","Type":"ContainerDied","Data":"17cb86137ff7430f5b55ff409a05190c1124d0b14efd3f6996bee0d25205a006"} Nov 03 17:58:45.673972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.673893 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" event={"ID":"2d343aa5-f3de-4b4f-adec-13c9d9bd4828","Type":"ContainerDied","Data":"61450a7bf684be57db2d9b538c6fa5fd4742da501b1659dc19fc10a73c8a97e4"} Nov 03 17:58:45.673972 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.673910 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61450a7bf684be57db2d9b538c6fa5fd4742da501b1659dc19fc10a73c8a97e4" Nov 03 17:58:45.686354 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.686330 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:45.823400 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823331 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\") pod \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " Nov 03 17:58:45.823400 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823380 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\") pod \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " Nov 03 17:58:45.823605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823436 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\") pod \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " Nov 03 17:58:45.823605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823494 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\") pod \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\" (UID: \"2d343aa5-f3de-4b4f-adec-13c9d9bd4828\") " Nov 03 17:58:45.823876 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823779 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume" pod "2d343aa5-f3de-4b4f-adec-13c9d9bd4828" (UID: "2d343aa5-f3de-4b4f-adec-13c9d9bd4828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:45.823992 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.823953 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume" pod "2d343aa5-f3de-4b4f-adec-13c9d9bd4828" (UID: "2d343aa5-f3de-4b4f-adec-13c9d9bd4828"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:45.825738 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.825711 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n" pod "2d343aa5-f3de-4b4f-adec-13c9d9bd4828" (UID: "2d343aa5-f3de-4b4f-adec-13c9d9bd4828"). InnerVolumeSpecName "kube-api-access-mwq6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:45.825897 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.825793 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials" pod "2d343aa5-f3de-4b4f-adec-13c9d9bd4828" (UID: "2d343aa5-f3de-4b4f-adec-13c9d9bd4828"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:45.924396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.924363 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:45.924396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.924391 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwq6n\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-kube-api-access-mwq6n\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:45.924396 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.924400 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:45.924642 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:45.924409 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2d343aa5-f3de-4b4f-adec-13c9d9bd4828-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:58:46.678372 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:46.678336 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz" Nov 03 17:58:46.879445 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:46.879405 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:46.930411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:46.930338 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-8c9b0f/payload-processing-external-model-e2e-extmod-b-8c9b0f-77b44ngbz"] Nov 03 17:58:47.660192 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:47.660160 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" path="/var/lib/kubelet/pods/2d343aa5-f3de-4b4f-adec-13c9d9bd4828/volumes" Nov 03 17:58:48.603578 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:58:48.603540 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:58:57.752281 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752241 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752559 4885 cpu_manager.go:414] "containerMap: removing container" podUID="078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" containerName="revoke-keys" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752572 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" containerName="revoke-keys" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752585 4885 cpu_manager.go:414] "containerMap: removing container" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerName="payload-processing" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752590 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerName="payload-processing" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752599 4885 cpu_manager.go:414] "containerMap: removing container" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752604 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752668 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" containerName="revoke-keys" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752678 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d343aa5-f3de-4b4f-adec-13c9d9bd4828" containerName="payload-processing" Nov 03 17:58:57.752795 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.752684 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ca5069-7166-4708-b3b0-9fd075788aec" containerName="payload-processing" Nov 03 17:58:57.757018 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.756986 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.811733 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.811700 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:58:57.820873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.820840 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.820873 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.820886 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdvj\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.821123 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.821038 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.821226 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.821205 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.860459 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.860426 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\"" Nov 03 17:58:57.860763 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.860742 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\"" Nov 03 17:58:57.860849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.860754 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\"" Nov 03 17:58:57.860849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.860795 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\"" Nov 03 17:58:57.860849 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.860804 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"e2e-cmp-legacy-d21aeb-provider-cred\"" Nov 03 17:58:57.862675 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.862660 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-4gwbm\"" Nov 03 17:58:57.922640 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.922611 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.922824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.922661 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.922824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.922688 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdvj\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.922824 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.922757 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.923458 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.923432 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.923576 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.923517 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.925770 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.925748 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:57.985153 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:57.985117 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdvj\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:58.069158 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.069056 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:58.468027 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.467984 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:58:58.475889 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:58:58.475863 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0254572c_af4d_417e_a4e3_f748c6d34fec.slice/crio-33b6944ebe347c17a3f0b1b2359a75fba455dbd41f6edb6b137ee5c5e26464c9 WatchSource:0}: Error finding container 33b6944ebe347c17a3f0b1b2359a75fba455dbd41f6edb6b137ee5c5e26464c9: Status 404 returned error can't find the container with id 33b6944ebe347c17a3f0b1b2359a75fba455dbd41f6edb6b137ee5c5e26464c9 Nov 03 17:58:58.730687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.730612 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" event={"ID":"0254572c-af4d-417e-a4e3-f748c6d34fec","Type":"ContainerStarted","Data":"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88"} Nov 03 17:58:58.730687 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.730661 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" event={"ID":"0254572c-af4d-417e-a4e3-f748c6d34fec","Type":"ContainerStarted","Data":"33b6944ebe347c17a3f0b1b2359a75fba455dbd41f6edb6b137ee5c5e26464c9"} Nov 03 17:58:58.730855 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.730781 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:58:58.851337 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:58.851281 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" podStartSLOduration=1.851266716 podStartE2EDuration="1.851266716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:58.849979885 +0000 UTC m=+2481.304668908" watchObservedRunningTime="2026-11-03 17:58:58.851266716 +0000 UTC m=+2481.305955734" Nov 03 17:58:59.074172 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:58:59.074083 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:59:09.738598 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:09.738569 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:59:18.604095 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:59:18.604045 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:59:19.221618 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:19.221580 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:59:35.304129 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.304095 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd"] Nov 03 17:59:35.307605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.307589 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.346255 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.346228 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.346391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.346280 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.346391 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.346308 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nsz\" (UniqueName: \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.361318 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.361290 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd"] Nov 03 17:59:35.411019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.410989 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:59:35.411019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.411011 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:59:35.411197 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.411019 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:59:35.412592 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.412578 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 17:59:35.446828 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.446802 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nsz\" (UniqueName: \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.446940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.446848 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.446940 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.446893 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.447570 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.447554 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.449244 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.449226 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.509214 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.509185 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nsz\" (UniqueName: \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\") pod \"maas-api-revoke-sub-d936894e-fctrd\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.619816 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.619790 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:35.984386 ip-10-0-129-76 kubenswrapper[4885]: W1103 17:59:35.984355 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5019da53_9cf7_424a_b5e3_a498554100fc.slice/crio-47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2 WatchSource:0}: Error finding container 47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2: Status 404 returned error can't find the container with id 47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2 Nov 03 17:59:35.986243 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:35.986221 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd"] Nov 03 17:59:36.893605 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:36.893575 4885 generic.go:356] "Generic (PLEG): container finished" podID="5019da53-9cf7-424a-b5e3-a498554100fc" containerID="6799b5f235d52ce4456b0816b02e331d27a0a84830b444cc5361977fbd25658c" exitCode=0 Nov 03 17:59:36.893978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:36.893668 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" event={"ID":"5019da53-9cf7-424a-b5e3-a498554100fc","Type":"ContainerDied","Data":"6799b5f235d52ce4456b0816b02e331d27a0a84830b444cc5361977fbd25658c"} Nov 03 17:59:36.893978 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:36.893720 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" event={"ID":"5019da53-9cf7-424a-b5e3-a498554100fc","Type":"ContainerStarted","Data":"47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2"} Nov 03 17:59:38.023529 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.023506 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:38.068794 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.068767 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\") pod \"5019da53-9cf7-424a-b5e3-a498554100fc\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " Nov 03 17:59:38.068918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.068826 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\") pod \"5019da53-9cf7-424a-b5e3-a498554100fc\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " Nov 03 17:59:38.068918 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.068870 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\" (UniqueName: \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\") pod \"5019da53-9cf7-424a-b5e3-a498554100fc\" (UID: \"5019da53-9cf7-424a-b5e3-a498554100fc\") " Nov 03 17:59:38.069223 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.069200 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca" pod "5019da53-9cf7-424a-b5e3-a498554100fc" (UID: "5019da53-9cf7-424a-b5e3-a498554100fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:38.070928 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.070905 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz" pod "5019da53-9cf7-424a-b5e3-a498554100fc" (UID: "5019da53-9cf7-424a-b5e3-a498554100fc"). InnerVolumeSpecName "kube-api-access-n7nsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:38.071019 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.070928 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca" pod "5019da53-9cf7-424a-b5e3-a498554100fc" (UID: "5019da53-9cf7-424a-b5e3-a498554100fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:59:38.169953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.169875 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:38.169953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.169901 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5019da53-9cf7-424a-b5e3-a498554100fc-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:38.169953 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.169912 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7nsz\" (UniqueName: \"kubernetes.io/projected/5019da53-9cf7-424a-b5e3-a498554100fc-kube-api-access-n7nsz\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:38.903869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.903838 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" Nov 03 17:59:38.903869 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.903858 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd" event={"ID":"5019da53-9cf7-424a-b5e3-a498554100fc","Type":"ContainerDied","Data":"47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2"} Nov 03 17:59:38.904163 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:38.903898 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47788db990096f6d61557d5ee792805f87f56b2320742b75c20cf8b3ed8048d2" Nov 03 17:59:48.409118 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:48.409087 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:59:48.611476 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:59:48.611434 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 17:59:51.936209 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:51.936176 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 17:59:53.409749 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.409697 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" podUID="0254572c-af4d-417e-a4e3-f748c6d34fec" containerName="payload-processing" containerID="cri-o://2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88" gracePeriod=25 Nov 03 17:59:53.668406 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.668351 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:59:53.809230 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809195 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\") pod \"0254572c-af4d-417e-a4e3-f748c6d34fec\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " Nov 03 17:59:53.809423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809257 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\") pod \"0254572c-af4d-417e-a4e3-f748c6d34fec\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " Nov 03 17:59:53.809423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809288 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\") pod \"0254572c-af4d-417e-a4e3-f748c6d34fec\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " Nov 03 17:59:53.809423 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809328 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\") pod \"0254572c-af4d-417e-a4e3-f748c6d34fec\" (UID: \"0254572c-af4d-417e-a4e3-f748c6d34fec\") " Nov 03 17:59:53.809710 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809680 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume" pod "0254572c-af4d-417e-a4e3-f748c6d34fec" (UID: "0254572c-af4d-417e-a4e3-f748c6d34fec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:53.809802 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.809740 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume" pod "0254572c-af4d-417e-a4e3-f748c6d34fec" (UID: "0254572c-af4d-417e-a4e3-f748c6d34fec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:53.811746 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.811719 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj" pod "0254572c-af4d-417e-a4e3-f748c6d34fec" (UID: "0254572c-af4d-417e-a4e3-f748c6d34fec"). InnerVolumeSpecName "kube-api-access-vjdvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:53.811844 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.811739 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials" pod "0254572c-af4d-417e-a4e3-f748c6d34fec" (UID: "0254572c-af4d-417e-a4e3-f748c6d34fec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:53.910646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.910610 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:53.910646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.910639 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdvj\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-kube-api-access-vjdvj\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:53.910646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.910648 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0254572c-af4d-417e-a4e3-f748c6d34fec-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:53.910646 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.910656 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0254572c-af4d-417e-a4e3-f748c6d34fec-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:59:53.972133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.972014 4885 generic.go:356] "Generic (PLEG): container finished" podID="0254572c-af4d-417e-a4e3-f748c6d34fec" containerID="2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88" exitCode=0 Nov 03 17:59:53.972133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.972089 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" Nov 03 17:59:53.972133 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.972095 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" event={"ID":"0254572c-af4d-417e-a4e3-f748c6d34fec","Type":"ContainerDied","Data":"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88"} Nov 03 17:59:53.972411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.972154 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt" event={"ID":"0254572c-af4d-417e-a4e3-f748c6d34fec","Type":"ContainerDied","Data":"33b6944ebe347c17a3f0b1b2359a75fba455dbd41f6edb6b137ee5c5e26464c9"} Nov 03 17:59:53.972411 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.972176 4885 scope.go:122] "RemoveContainer" containerID="2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88" Nov 03 17:59:53.981026 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.981008 4885 scope.go:122] "RemoveContainer" containerID="2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88" Nov 03 17:59:53.981340 ip-10-0-129-76 kubenswrapper[4885]: E1103 17:59:53.981315 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88\": container with ID starting with 2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88 not found: ID does not exist" containerID="2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88" Nov 03 17:59:53.981420 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:53.981351 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88"} err="failed to get container status \"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88\": rpc error: code = NotFound desc = could not find container \"2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88\": container with ID starting with 2da565bb7f9f8c9004b9db53bef3957572e8cf2c59bcd15bc2e175673d637e88 not found: ID does not exist" Nov 03 17:59:54.140556 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:54.140512 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:59:54.144683 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:54.144661 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-dc94pwkpt"] Nov 03 17:59:55.660981 ip-10-0-129-76 kubenswrapper[4885]: I1103 17:59:55.660940 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0254572c-af4d-417e-a4e3-f748c6d34fec" path="/var/lib/kubelet/pods/0254572c-af4d-417e-a4e3-f748c6d34fec/volumes" Nov 03 18:00:00.158517 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158437 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158757 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0254572c-af4d-417e-a4e3-f748c6d34fec" containerName="payload-processing" Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158767 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0254572c-af4d-417e-a4e3-f748c6d34fec" containerName="payload-processing" Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158786 4885 cpu_manager.go:414] "containerMap: removing container" podUID="5019da53-9cf7-424a-b5e3-a498554100fc" containerName="revoke-keys" Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158790 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5019da53-9cf7-424a-b5e3-a498554100fc" containerName="revoke-keys" Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158843 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0254572c-af4d-417e-a4e3-f748c6d34fec" containerName="payload-processing" Nov 03 18:00:00.158914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.158851 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="5019da53-9cf7-424a-b5e3-a498554100fc" containerName="revoke-keys" Nov 03 18:00:00.163480 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.163457 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.165147 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.165123 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:00.168313 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.168294 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.217467 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.217443 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:00.217607 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.217443 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:00.217607 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.217531 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 18:00:00.268053 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.268013 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6r7\" (UniqueName: \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.268239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.268060 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.268239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.268189 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq2n\" (UniqueName: \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.268239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.268226 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.270408 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.270374 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:00.271763 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.271731 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:00.368805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.368778 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.368917 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.368814 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6r7\" (UniqueName: \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.368917 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.368855 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.369133 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.368916 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq2n\" (UniqueName: \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.369502 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.369485 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.369541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.369499 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.431825 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.431748 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq2n\" (UniqueName: \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.431825 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.431748 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6r7\" (UniqueName: \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\") pod \"maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.475589 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.475556 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:00.482705 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.482680 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:00.857855 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:00:00.857829 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141849bb_2a04_4ae3_8e82_399d4aec4789.slice/crio-ef5e050591edd30f920d77af87f13276429ac73f0ef247e5d118d18e8328d00e WatchSource:0}: Error finding container ef5e050591edd30f920d77af87f13276429ac73f0ef247e5d118d18e8328d00e: Status 404 returned error can't find the container with id ef5e050591edd30f920d77af87f13276429ac73f0ef247e5d118d18e8328d00e Nov 03 18:00:00.859886 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.859860 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:00.905047 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:00:00.905022 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb467bdc_b057_493e_ad2f_66ba6689f4b5.slice/crio-736a76a6f4eba0ff150c8161baf53f71150567341f74f245dcf4b0d9eaf4a2c6 WatchSource:0}: Error finding container 736a76a6f4eba0ff150c8161baf53f71150567341f74f245dcf4b0d9eaf4a2c6: Status 404 returned error can't find the container with id 736a76a6f4eba0ff150c8161baf53f71150567341f74f245dcf4b0d9eaf4a2c6 Nov 03 18:00:00.907082 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:00.907048 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:01.003316 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:01.003270 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerStarted","Data":"ef5e050591edd30f920d77af87f13276429ac73f0ef247e5d118d18e8328d00e"} Nov 03 18:00:01.004690 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:01.004646 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerStarted","Data":"736a76a6f4eba0ff150c8161baf53f71150567341f74f245dcf4b0d9eaf4a2c6"} Nov 03 18:00:02.011123 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.011090 4885 generic.go:356] "Generic (PLEG): container finished" podID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerID="d3a6ccb02c918b31eaf0776cc4a4998b161d8aa6b35cedeb52c00843c4555941" exitCode=60 Nov 03 18:00:02.011601 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.011183 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerDied","Data":"d3a6ccb02c918b31eaf0776cc4a4998b161d8aa6b35cedeb52c00843c4555941"} Nov 03 18:00:02.011601 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.011378 4885 scope.go:122] "RemoveContainer" containerID="d3a6ccb02c918b31eaf0776cc4a4998b161d8aa6b35cedeb52c00843c4555941" Nov 03 18:00:02.012732 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.012715 4885 generic.go:356] "Generic (PLEG): container finished" podID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerID="431b09c85cc22b61cd40e6e831e4a9f06de04a9b0cbd800036c3fe0a51128c59" exitCode=60 Nov 03 18:00:02.012792 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.012755 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerDied","Data":"431b09c85cc22b61cd40e6e831e4a9f06de04a9b0cbd800036c3fe0a51128c59"} Nov 03 18:00:02.012946 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:02.012929 4885 scope.go:122] "RemoveContainer" containerID="431b09c85cc22b61cd40e6e831e4a9f06de04a9b0cbd800036c3fe0a51128c59" Nov 03 18:00:03.020043 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.020009 4885 generic.go:356] "Generic (PLEG): container finished" podID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerID="1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580" exitCode=60 Nov 03 18:00:03.020542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.020161 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerDied","Data":"1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580"} Nov 03 18:00:03.020542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.020229 4885 scope.go:122] "RemoveContainer" containerID="431b09c85cc22b61cd40e6e831e4a9f06de04a9b0cbd800036c3fe0a51128c59" Nov 03 18:00:03.020542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.020487 4885 scope.go:122] "RemoveContainer" containerID="1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580" Nov 03 18:00:03.020736 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:03.020717 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m_odh-ai-gateway-infra(141849bb-2a04-4ae3-8e82-399d4aec4789)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" Nov 03 18:00:03.022493 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.022473 4885 generic.go:356] "Generic (PLEG): container finished" podID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerID="3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76" exitCode=60 Nov 03 18:00:03.022621 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.022502 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerDied","Data":"3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76"} Nov 03 18:00:03.022776 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.022760 4885 scope.go:122] "RemoveContainer" containerID="3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76" Nov 03 18:00:03.022980 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:03.022954 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f_odh-ai-gateway-infra(bb467bdc-b057-493e-ad2f-66ba6689f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" Nov 03 18:00:03.032700 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.032676 4885 scope.go:122] "RemoveContainer" containerID="d3a6ccb02c918b31eaf0776cc4a4998b161d8aa6b35cedeb52c00843c4555941" Nov 03 18:00:03.660171 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:03.660139 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:00:04.028059 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:04.027978 4885 scope.go:122] "RemoveContainer" containerID="3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76" Nov 03 18:00:04.028481 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:04.028219 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f_odh-ai-gateway-infra(bb467bdc-b057-493e-ad2f-66ba6689f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" Nov 03 18:00:04.029452 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:04.029435 4885 scope.go:122] "RemoveContainer" containerID="1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580" Nov 03 18:00:04.029617 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:04.029602 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m_odh-ai-gateway-infra(141849bb-2a04-4ae3-8e82-399d4aec4789)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" Nov 03 18:00:13.656281 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:13.656252 4885 scope.go:122] "RemoveContainer" containerID="3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76" Nov 03 18:00:14.075455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:14.075366 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerStarted","Data":"87945a41ffdd08e7ce4bd0f747ab1110530c2bbb486e0575165cc56251019bb1"} Nov 03 18:00:14.195230 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:14.195174 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" podStartSLOduration=14.195161437 podStartE2EDuration="14.195161437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:14.193851982 +0000 UTC m=+2556.648541006" watchObservedRunningTime="2026-11-03 18:00:14.195161437 +0000 UTC m=+2556.649850455" Nov 03 18:00:14.655473 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:14.655397 4885 scope.go:122] "RemoveContainer" containerID="1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580" Nov 03 18:00:15.082536 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.082449 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerStarted","Data":"5aa2336204bd13f171bfcabea38d55d471da1c7120473f8df14e44894c22b51d"} Nov 03 18:00:15.084318 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.084293 4885 generic.go:356] "Generic (PLEG): container finished" podID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerID="87945a41ffdd08e7ce4bd0f747ab1110530c2bbb486e0575165cc56251019bb1" exitCode=60 Nov 03 18:00:15.084426 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.084322 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerDied","Data":"87945a41ffdd08e7ce4bd0f747ab1110530c2bbb486e0575165cc56251019bb1"} Nov 03 18:00:15.084426 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.084357 4885 scope.go:122] "RemoveContainer" containerID="3870a2c0871f452d59590b56bf2e54248cfea085f85e257bf875b2dd9c263d76" Nov 03 18:00:15.084704 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.084690 4885 scope.go:122] "RemoveContainer" containerID="87945a41ffdd08e7ce4bd0f747ab1110530c2bbb486e0575165cc56251019bb1" Nov 03 18:00:15.084892 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:15.084870 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f_odh-ai-gateway-infra(bb467bdc-b057-493e-ad2f-66ba6689f4b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" Nov 03 18:00:15.201813 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.201757 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" podStartSLOduration=15.201740451 podStartE2EDuration="15.201740451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:15.200566737 +0000 UTC m=+2557.655255754" watchObservedRunningTime="2026-11-03 18:00:15.201740451 +0000 UTC m=+2557.656429475" Nov 03 18:00:15.206536 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:15.206508 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:16.093420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.093003 4885 generic.go:356] "Generic (PLEG): container finished" podID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerID="5aa2336204bd13f171bfcabea38d55d471da1c7120473f8df14e44894c22b51d" exitCode=60 Nov 03 18:00:16.093420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.093133 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerDied","Data":"5aa2336204bd13f171bfcabea38d55d471da1c7120473f8df14e44894c22b51d"} Nov 03 18:00:16.093420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.093183 4885 scope.go:122] "RemoveContainer" containerID="1d45f5f174e1213a50c2c51759a14870652364adaa2a68d8e13be8a579e5c580" Nov 03 18:00:16.093918 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.093513 4885 scope.go:122] "RemoveContainer" containerID="5aa2336204bd13f171bfcabea38d55d471da1c7120473f8df14e44894c22b51d" Nov 03 18:00:16.093918 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:16.093741 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m_odh-ai-gateway-infra(141849bb-2a04-4ae3-8e82-399d4aec4789)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" Nov 03 18:00:16.213746 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.213718 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:16.222573 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.222553 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:16.298474 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.298445 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\") pod \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " Nov 03 18:00:16.298645 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.298502 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\" (UniqueName: \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\") pod \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\" (UID: \"bb467bdc-b057-493e-ad2f-66ba6689f4b5\") " Nov 03 18:00:16.298821 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.298803 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca" pod "bb467bdc-b057-493e-ad2f-66ba6689f4b5" (UID: "bb467bdc-b057-493e-ad2f-66ba6689f4b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:16.300636 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.300607 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7" pod "bb467bdc-b057-493e-ad2f-66ba6689f4b5" (UID: "bb467bdc-b057-493e-ad2f-66ba6689f4b5"). InnerVolumeSpecName "kube-api-access-sx6r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:16.399265 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.399235 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb467bdc-b057-493e-ad2f-66ba6689f4b5-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:16.399265 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:16.399261 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx6r7\" (UniqueName: \"kubernetes.io/projected/bb467bdc-b057-493e-ad2f-66ba6689f4b5-kube-api-access-sx6r7\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:17.099028 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.098996 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" Nov 03 18:00:17.099028 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.099003 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f" event={"ID":"bb467bdc-b057-493e-ad2f-66ba6689f4b5","Type":"ContainerDied","Data":"736a76a6f4eba0ff150c8161baf53f71150567341f74f245dcf4b0d9eaf4a2c6"} Nov 03 18:00:17.099571 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.099124 4885 scope.go:122] "RemoveContainer" containerID="87945a41ffdd08e7ce4bd0f747ab1110530c2bbb486e0575165cc56251019bb1" Nov 03 18:00:17.247979 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.247950 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:17.269537 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.269510 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:17.320558 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.320516 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-029ca0-29895480-nmz7f"] Nov 03 18:00:17.406459 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.406429 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\" (UniqueName: \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\") pod \"141849bb-2a04-4ae3-8e82-399d4aec4789\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " Nov 03 18:00:17.406640 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.406477 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\") pod \"141849bb-2a04-4ae3-8e82-399d4aec4789\" (UID: \"141849bb-2a04-4ae3-8e82-399d4aec4789\") " Nov 03 18:00:17.406836 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.406813 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca" pod "141849bb-2a04-4ae3-8e82-399d4aec4789" (UID: "141849bb-2a04-4ae3-8e82-399d4aec4789"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:17.408744 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.408713 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n" pod "141849bb-2a04-4ae3-8e82-399d4aec4789" (UID: "141849bb-2a04-4ae3-8e82-399d4aec4789"). InnerVolumeSpecName "kube-api-access-mzq2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:17.507807 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.507777 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141849bb-2a04-4ae3-8e82-399d4aec4789-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:17.507807 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.507805 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq2n\" (UniqueName: \"kubernetes.io/projected/141849bb-2a04-4ae3-8e82-399d4aec4789-kube-api-access-mzq2n\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:17.661531 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.661444 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" path="/var/lib/kubelet/pods/bb467bdc-b057-493e-ad2f-66ba6689f4b5/volumes" Nov 03 18:00:17.687105 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:17.687056 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:00:18.108934 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:18.108843 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" event={"ID":"141849bb-2a04-4ae3-8e82-399d4aec4789","Type":"ContainerDied","Data":"ef5e050591edd30f920d77af87f13276429ac73f0ef247e5d118d18e8328d00e"} Nov 03 18:00:18.108934 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:18.108915 4885 scope.go:122] "RemoveContainer" containerID="5aa2336204bd13f171bfcabea38d55d471da1c7120473f8df14e44894c22b51d" Nov 03 18:00:18.109493 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:18.108859 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m" Nov 03 18:00:18.278423 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:18.278391 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:18.282505 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:18.282476 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895480-qtr7m"] Nov 03 18:00:18.603992 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:18.603958 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:00:19.661463 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:19.661423 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" path="/var/lib/kubelet/pods/141849bb-2a04-4ae3-8e82-399d4aec4789/volumes" Nov 03 18:00:26.580791 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.580756 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581107 4885 cpu_manager.go:414] "containerMap: removing container" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581119 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581132 4885 cpu_manager.go:414] "containerMap: removing container" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581141 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581155 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581160 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581167 4885 cpu_manager.go:414] "containerMap: removing container" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581172 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581178 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581182 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581499 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581242 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581499 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581250 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581499 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581255 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.581499 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581260 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:00:26.581499 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.581264 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:00:26.585676 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.585659 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.638905 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.638875 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:00:26.674904 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.674876 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.674904 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.674908 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.675170 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.674941 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.675170 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.675102 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrwb\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.688541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688516 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"e2e-cfl-d35f0c-provider-cred\"" Nov 03 18:00:26.688541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688527 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"openshift-service-ca.crt\"" Nov 03 18:00:26.688541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688542 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"agc-cred-efacdc6a26da\"" Nov 03 18:00:26.688795 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688527 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"kube-root-ca.crt\"" Nov 03 18:00:26.688795 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688573 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"routing-overlay\"" Nov 03 18:00:26.688795 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.688604 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-029ca0\"" Nov 03 18:00:26.693031 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.693016 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-029ca0\"/\"payload-processing-external-model-e2e--58d254fa-dockercfg-zmxcc\"" Nov 03 18:00:26.775872 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.775824 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrwb\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.776110 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.775892 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.776110 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.775922 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.776110 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.775968 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.776740 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.776712 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.777013 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.776991 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.779366 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.779341 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.838636 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.838560 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrwb\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\") pod \"payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:26.898109 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:26.898077 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:27.273705 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:00:27.273668 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d1e29_6504_4d0a_9012_7a5303191598.slice/crio-3aa6d8886c2119e7b5d1b34665c64d4c295ef429257478e77daec6c8ec9bac26 WatchSource:0}: Error finding container 3aa6d8886c2119e7b5d1b34665c64d4c295ef429257478e77daec6c8ec9bac26: Status 404 returned error can't find the container with id 3aa6d8886c2119e7b5d1b34665c64d4c295ef429257478e77daec6c8ec9bac26 Nov 03 18:00:27.275558 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:27.275482 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:00:27.618244 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:27.618206 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:00:28.156770 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:28.156739 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" event={"ID":"903d1e29-6504-4d0a-9012-7a5303191598","Type":"ContainerStarted","Data":"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2"} Nov 03 18:00:28.156966 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:28.156795 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" event={"ID":"903d1e29-6504-4d0a-9012-7a5303191598","Type":"ContainerStarted","Data":"3aa6d8886c2119e7b5d1b34665c64d4c295ef429257478e77daec6c8ec9bac26"} Nov 03 18:00:28.156966 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:28.156819 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:28.275931 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:28.275869 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" podStartSLOduration=2.275852768 podStartE2EDuration="2.275852768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:28.275529628 +0000 UTC m=+2570.730218652" watchObservedRunningTime="2026-11-03 18:00:28.275852768 +0000 UTC m=+2570.730541794" Nov 03 18:00:39.165081 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:39.165037 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:00:39.562648 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:00:39.562551 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:00:48.603896 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:00:48.603855 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:01:18.605121 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:01:18.605082 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:01:48.604852 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:01:48.604809 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:02:18.608763 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:02:18.608724 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:02:26.050506 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:26.050467 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz"] Nov 03 18:02:26.101453 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:26.101417 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e87bf8a-twfxz"] Nov 03 18:02:27.661040 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:27.661005 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3383005-59b2-45c3-abf9-21e5b7c528b6" path="/var/lib/kubelet/pods/c3383005-59b2-45c3-abf9-21e5b7c528b6/volumes" Nov 03 18:02:38.109123 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:38.109084 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg"] Nov 03 18:02:38.160496 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:38.160459 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67ac462-v56rg"] Nov 03 18:02:39.660711 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:39.660680 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8d9df1-cb35-4474-b6c9-abd29efea935" path="/var/lib/kubelet/pods/9d8d9df1-cb35-4474-b6c9-abd29efea935/volumes" Nov 03 18:02:48.608402 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:02:48.608362 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:02:50.054777 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:50.054729 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8"] Nov 03 18:02:50.105027 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:50.104988 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac7c7814-wn6g8"] Nov 03 18:02:50.460483 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:50.460453 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:02:51.660642 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:02:51.660606 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fce8d73-3572-4451-8a32-30f0f7e57662" path="/var/lib/kubelet/pods/7fce8d73-3572-4451-8a32-30f0f7e57662/volumes" Nov 03 18:03:05.562387 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:05.562289 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:03:06.049861 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:06.049827 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95"] Nov 03 18:03:06.052708 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:06.052679 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bc1f8d-xxd95"] Nov 03 18:03:07.660474 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:07.660438 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6021d689-c344-4996-81b5-46cfc4d2cec1" path="/var/lib/kubelet/pods/6021d689-c344-4996-81b5-46cfc4d2cec1/volumes" Nov 03 18:03:10.479727 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.479689 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:03:10.563541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.563468 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" podUID="903d1e29-6504-4d0a-9012-7a5303191598" containerName="payload-processing" containerID="cri-o://d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2" gracePeriod=25 Nov 03 18:03:10.806186 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.806160 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:03:10.877950 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.877912 4885 generic.go:356] "Generic (PLEG): container finished" podID="903d1e29-6504-4d0a-9012-7a5303191598" containerID="d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2" exitCode=0 Nov 03 18:03:10.878152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.877959 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" Nov 03 18:03:10.878152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.878007 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" event={"ID":"903d1e29-6504-4d0a-9012-7a5303191598","Type":"ContainerDied","Data":"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2"} Nov 03 18:03:10.878152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.878060 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv" event={"ID":"903d1e29-6504-4d0a-9012-7a5303191598","Type":"ContainerDied","Data":"3aa6d8886c2119e7b5d1b34665c64d4c295ef429257478e77daec6c8ec9bac26"} Nov 03 18:03:10.878152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.878102 4885 scope.go:122] "RemoveContainer" containerID="d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2" Nov 03 18:03:10.886857 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.886834 4885 scope.go:122] "RemoveContainer" containerID="d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2" Nov 03 18:03:10.887176 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:03:10.887152 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2\": container with ID starting with d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2 not found: ID does not exist" containerID="d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2" Nov 03 18:03:10.887232 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.887186 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2"} err="failed to get container status \"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2\": rpc error: code = NotFound desc = could not find container \"d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2\": container with ID starting with d60e3c60408e60e5ff525a514cbbe1d16063cb286343685abb137dc48b290cf2 not found: ID does not exist" Nov 03 18:03:10.924946 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.924911 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\") pod \"903d1e29-6504-4d0a-9012-7a5303191598\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " Nov 03 18:03:10.925128 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.924950 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\") pod \"903d1e29-6504-4d0a-9012-7a5303191598\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " Nov 03 18:03:10.925128 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925033 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\") pod \"903d1e29-6504-4d0a-9012-7a5303191598\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " Nov 03 18:03:10.925257 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925162 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\") pod \"903d1e29-6504-4d0a-9012-7a5303191598\" (UID: \"903d1e29-6504-4d0a-9012-7a5303191598\") " Nov 03 18:03:10.925403 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925356 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume" pod "903d1e29-6504-4d0a-9012-7a5303191598" (UID: "903d1e29-6504-4d0a-9012-7a5303191598"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:10.925539 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925460 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume" pod "903d1e29-6504-4d0a-9012-7a5303191598" (UID: "903d1e29-6504-4d0a-9012-7a5303191598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:10.925800 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925777 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:03:10.926030 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.925802 4885 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/903d1e29-6504-4d0a-9012-7a5303191598-routing-overlay-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:03:10.927590 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.927562 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb" pod "903d1e29-6504-4d0a-9012-7a5303191598" (UID: "903d1e29-6504-4d0a-9012-7a5303191598"). InnerVolumeSpecName "kube-api-access-gbrwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:10.928262 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:10.928239 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials" pod "903d1e29-6504-4d0a-9012-7a5303191598" (UID: "903d1e29-6504-4d0a-9012-7a5303191598"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:11.026862 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:11.026826 4885 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-provider-credentials\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:03:11.026862 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:11.026860 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrwb\" (UniqueName: \"kubernetes.io/projected/903d1e29-6504-4d0a-9012-7a5303191598-kube-api-access-gbrwb\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:03:11.300817 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:11.300779 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:03:11.305272 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:11.305248 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-029ca0/payload-processing-external-model-e2e-extmod-a-029ca0-86f5bcnbv"] Nov 03 18:03:11.661345 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:11.661309 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903d1e29-6504-4d0a-9012-7a5303191598" path="/var/lib/kubelet/pods/903d1e29-6504-4d0a-9012-7a5303191598/volumes" Nov 03 18:03:18.607843 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:03:18.607802 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:03:19.058399 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:19.058355 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn"] Nov 03 18:03:19.109403 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:19.109356 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec87d6fd-2s8wn"] Nov 03 18:03:19.660849 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:19.660814 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c4c8d4-c6e1-4896-b6dc-8792960c7b35" path="/var/lib/kubelet/pods/a5c4c8d4-c6e1-4896-b6dc-8792960c7b35/volumes" Nov 03 18:03:22.755895 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:22.755861 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:03:32.051151 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:32.051110 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq"] Nov 03 18:03:32.102325 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:32.102284 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6b1784a-ghcqq"] Nov 03 18:03:33.661402 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:33.661367 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078ef6e5-eb34-4c63-95cd-b37eb00b7b9f" path="/var/lib/kubelet/pods/078ef6e5-eb34-4c63-95cd-b37eb00b7b9f/volumes" Nov 03 18:03:38.550753 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:38.550713 4885 scope.go:122] "RemoveContainer" containerID="b324b5c52c39c09e41b7e5dc0efceb83f5e176cc2d861deb15e59077e3204d1f" Nov 03 18:03:38.559165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:03:38.559137 4885 scope.go:122] "RemoveContainer" containerID="85c3843e8058a9dc594478de2ec2802d2dcb6fed75093fa2bb22e17f47b62856" Nov 03 18:03:48.606338 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:03:48.606290 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:04:18.604515 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:04:18.604476 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:04:38.586278 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:38.586204 4885 scope.go:122] "RemoveContainer" containerID="29f96ab9d2df944372f234bd16a0274566c134af626d4203f9e50b2138184112" Nov 03 18:04:38.594467 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:38.594444 4885 scope.go:122] "RemoveContainer" containerID="5589de9dd04be6c3efd771b573ea6b0d519e344bc3bcd316bda513f7ea0a2673" Nov 03 18:04:38.601670 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:38.601648 4885 scope.go:122] "RemoveContainer" containerID="17cb86137ff7430f5b55ff409a05190c1124d0b14efd3f6996bee0d25205a006" Nov 03 18:04:38.609140 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:38.609108 4885 scope.go:122] "RemoveContainer" containerID="32b415052e63b0348c93186becbaed2e58f8bb69a71039ec696f1872e57a76d0" Nov 03 18:04:38.617172 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:38.617155 4885 scope.go:122] "RemoveContainer" containerID="84f1c19c49d207670d6eebd2a4b24daf3101c783ea1866d6ad39a3bc27c79fb2" Nov 03 18:04:39.055468 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:39.055432 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd"] Nov 03 18:04:39.106608 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:39.106576 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d936894e-fctrd"] Nov 03 18:04:39.660971 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:04:39.660934 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5019da53-9cf7-424a-b5e3-a498554100fc" path="/var/lib/kubelet/pods/5019da53-9cf7-424a-b5e3-a498554100fc/volumes" Nov 03 18:04:48.605963 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:04:48.605922 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:05:18.605484 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:05:18.605441 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:05:31.866456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:05:31.866409 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:05:38.653812 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:05:38.653772 4885 scope.go:122] "RemoveContainer" containerID="6799b5f235d52ce4456b0816b02e331d27a0a84830b444cc5361977fbd25658c" Nov 03 18:05:46.948465 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:05:46.948432 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:05:48.605034 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:05:48.604994 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:06:18.605886 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:06:18.605801 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:06:48.606150 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:06:48.606110 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:07:18.605120 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:07:18.605058 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:07:48.605493 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:07:48.605455 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:08:18.604950 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:08:18.604908 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:08:48.605363 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:08:48.605315 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:09:18.606183 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:09:18.606102 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:09:48.605787 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:09:48.605745 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:10:18.606003 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:10:18.605882 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:10:48.605137 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:10:48.605098 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:11:18.604882 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:11:18.604835 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:11:48.605649 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:11:48.605613 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:12:18.604784 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:12:18.604700 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:12:27.718331 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:12:27.718275 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:12:40.368646 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:12:40.368604 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:12:48.605374 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:12:48.605339 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:12:53.052971 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:12:53.052933 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:13:06.718006 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:13:06.717970 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:13:18.605169 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:13:18.605127 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:13:23.059906 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:13:23.059870 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:13:39.476613 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:13:39.476517 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:13:48.604711 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:13:48.604668 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:13:59.548956 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:13:59.548918 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:14:15.367354 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367316 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367668 4885 cpu_manager.go:414] "containerMap: removing container" podUID="903d1e29-6504-4d0a-9012-7a5303191598" containerName="payload-processing" Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367680 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903d1e29-6504-4d0a-9012-7a5303191598" containerName="payload-processing" Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367690 4885 cpu_manager.go:414] "containerMap: removing container" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367695 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb467bdc-b057-493e-ad2f-66ba6689f4b5" containerName="cleanup" Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367764 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="141849bb-2a04-4ae3-8e82-399d4aec4789" containerName="cleanup" Nov 03 18:14:15.367851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.367773 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="903d1e29-6504-4d0a-9012-7a5303191598" containerName="payload-processing" Nov 03 18:14:15.370675 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.370655 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.409994 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.409960 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.409994 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.409992 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410015 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410099 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5wx\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410190 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410225 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410465 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410290 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410465 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410317 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.410465 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.410373 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.424202 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.424175 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\"" Nov 03 18:14:15.481842 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.481808 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:14:15.510967 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.510930 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.510981 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511010 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511039 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511124 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511152 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511494 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511189 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511494 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511231 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5wx\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511494 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511259 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511494 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511402 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511707 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511585 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511707 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511625 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511707 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511678 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.511832 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.511809 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.513714 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.513689 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.513817 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.513775 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.573884 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.573849 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.574163 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.574142 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5wx\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\") pod \"maas-default-gateway-openshift-default-58c6585447-qqzc9\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:15.683185 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:15.683152 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:16.098712 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:14:16.098680 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67af5e29_dc79_419b_9e03_2fb871b2de27.slice/crio-8e2773aea57e94632e5b3cb4c62bed81f65eed126276d57010cc10ca75ec207e WatchSource:0}: Error finding container 8e2773aea57e94632e5b3cb4c62bed81f65eed126276d57010cc10ca75ec207e: Status 404 returned error can't find the container with id 8e2773aea57e94632e5b3cb4c62bed81f65eed126276d57010cc10ca75ec207e Nov 03 18:14:16.099056 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.099034 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:14:16.100811 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.100781 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:16.100893 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.100844 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:16.100893 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.100874 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:16.641728 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.641689 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" event={"ID":"67af5e29-dc79-419b-9e03-2fb871b2de27","Type":"ContainerStarted","Data":"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff"} Nov 03 18:14:16.642153 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.641747 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" event={"ID":"67af5e29-dc79-419b-9e03-2fb871b2de27","Type":"ContainerStarted","Data":"8e2773aea57e94632e5b3cb4c62bed81f65eed126276d57010cc10ca75ec207e"} Nov 03 18:14:16.683736 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.683707 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:16.767555 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:16.767491 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podStartSLOduration=1.767473169 podStartE2EDuration="1.767473169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:16.765700672 +0000 UTC m=+3399.220389695" watchObservedRunningTime="2026-11-03 18:14:16.767473169 +0000 UTC m=+3399.222162195" Nov 03 18:14:17.684802 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:17.684770 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:17.685205 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:17.684830 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:18.635451 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:14:18.635406 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:14:18.684442 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:18.684404 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:18.684672 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:18.684467 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:19.684128 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:19.684092 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:19.684532 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:19.684149 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:20.684451 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:20.684414 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:20.684944 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:20.684480 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:21.683938 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:21.683899 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:21.684140 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:21.683958 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:22.683760 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:22.683725 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:22.684191 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:22.683784 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:23.684468 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:23.684430 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qqzc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:14:23.684876 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:23.684495 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 18:14:25.536829 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:25.536798 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:25.537274 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:25.537120 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:25.537951 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:25.537932 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:14:27.265535 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:27.265500 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:14:48.607227 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:14:48.607187 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:14:53.052887 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:14:53.052853 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:15:00.168265 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.168167 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:00.172162 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.172142 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.225890 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.225844 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:15:00.243251 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.243218 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.243394 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.243255 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7xc\" (UniqueName: \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.277318 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.277275 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 18:15:00.277529 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.277509 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:15:00.278668 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.278647 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:00.343964 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.343924 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.343964 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.343967 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7xc\" (UniqueName: \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.344742 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.344720 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.357619 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.357584 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:15:00.361129 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.361109 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.395350 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.395329 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:15:00.398466 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.398450 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.407374 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.407351 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7xc\" (UniqueName: \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\") pod \"maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.422523 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.420748 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:15:00.444934 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.444903 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445120 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.444945 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445120 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.444974 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445120 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445053 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445248 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445120 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445248 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445213 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6bt\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445327 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445273 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445381 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445363 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445441 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445389 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445441 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445419 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445554 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445451 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445554 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445478 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445554 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445503 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445565 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445610 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.445751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445642 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2cmc\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445680 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.445751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.445724 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.485586 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.485542 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:00.486670 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.486545 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:15:00.547405 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547375 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547556 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547442 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547556 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547468 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547556 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547494 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547556 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547520 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547556 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547546 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547571 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547605 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547636 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547665 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2cmc\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547709 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547753 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547786 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547819 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547849 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547873 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.547930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547915 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.548518 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.547947 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6bt\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.548595 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.548543 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.551261 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.549058 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.551261 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.550005 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.551261 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.550108 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.551261 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.550601 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.551261 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.550889 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.551666 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.551457 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.553220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.552537 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.553220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.552813 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.553220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.553052 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.553820 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.553802 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.562424 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.562393 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.564965 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.564938 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.565210 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.565191 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.624825 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.624790 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6bt\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.625125 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.625101 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.625197 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.625108 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8nps7\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.632004 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.631982 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2cmc\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\") pod \"maas-default-gateway-openshift-default-58c6585447-td42d\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.673834 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.673761 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:00.710925 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.710891 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:00.860666 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:15:00.860632 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010173d7_e198_4520_bd04_c779b36e1791.slice/crio-98862db4dcb99a537700f783e75721934bd9d7331f672834f3fc0dba6d0ee085 WatchSource:0}: Error finding container 98862db4dcb99a537700f783e75721934bd9d7331f672834f3fc0dba6d0ee085: Status 404 returned error can't find the container with id 98862db4dcb99a537700f783e75721934bd9d7331f672834f3fc0dba6d0ee085 Nov 03 18:15:00.862136 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:00.862114 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:01.050355 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.050333 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:15:01.100244 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.100214 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:15:01.103120 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:15:01.103089 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3bb0a_9601_4f3e_bd88_67ab746c9258.slice/crio-dd97414b8b269548d5167d17f58df85613f7b089c634557eb0a59ef4725be08d WatchSource:0}: Error finding container dd97414b8b269548d5167d17f58df85613f7b089c634557eb0a59ef4725be08d: Status 404 returned error can't find the container with id dd97414b8b269548d5167d17f58df85613f7b089c634557eb0a59ef4725be08d Nov 03 18:15:01.103748 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:15:01.103722 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f32ece_3f23_4c8c_85b3_ec6f95e92019.slice/crio-d0f21222b790ac057f5dd0ff41962d6b3ce3b6add78fb69925087ef54c6c44c9 WatchSource:0}: Error finding container d0f21222b790ac057f5dd0ff41962d6b3ce3b6add78fb69925087ef54c6c44c9: Status 404 returned error can't find the container with id d0f21222b790ac057f5dd0ff41962d6b3ce3b6add78fb69925087ef54c6c44c9 Nov 03 18:15:01.105652 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.105621 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.105742 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.105691 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.105742 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.105728 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.106239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.106158 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.106239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.106229 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.106369 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.106273 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:01.849115 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.848615 4885 generic.go:356] "Generic (PLEG): container finished" podID="010173d7-e198-4520-bd04-c779b36e1791" containerID="d7301031b0366c80443ac5e80ac19f521192150e2414ef7f39524a37fd8f4354" exitCode=60 Nov 03 18:15:01.849115 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.848742 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerDied","Data":"d7301031b0366c80443ac5e80ac19f521192150e2414ef7f39524a37fd8f4354"} Nov 03 18:15:01.849115 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.848797 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerStarted","Data":"98862db4dcb99a537700f783e75721934bd9d7331f672834f3fc0dba6d0ee085"} Nov 03 18:15:01.860111 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.849861 4885 scope.go:122] "RemoveContainer" containerID="d7301031b0366c80443ac5e80ac19f521192150e2414ef7f39524a37fd8f4354" Nov 03 18:15:01.863275 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.863235 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" event={"ID":"aba3bb0a-9601-4f3e-bd88-67ab746c9258","Type":"ContainerStarted","Data":"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c"} Nov 03 18:15:01.863437 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.863308 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" event={"ID":"aba3bb0a-9601-4f3e-bd88-67ab746c9258","Type":"ContainerStarted","Data":"dd97414b8b269548d5167d17f58df85613f7b089c634557eb0a59ef4725be08d"} Nov 03 18:15:01.866702 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.866587 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" event={"ID":"03f32ece-3f23-4c8c-85b3-ec6f95e92019","Type":"ContainerStarted","Data":"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae"} Nov 03 18:15:01.866702 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:01.866630 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" event={"ID":"03f32ece-3f23-4c8c-85b3-ec6f95e92019","Type":"ContainerStarted","Data":"d0f21222b790ac057f5dd0ff41962d6b3ce3b6add78fb69925087ef54c6c44c9"} Nov 03 18:15:02.244093 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.243683 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podStartSLOduration=2.243658504 podStartE2EDuration="2.243658504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.241524952 +0000 UTC m=+3444.696213977" watchObservedRunningTime="2026-11-03 18:15:02.243658504 +0000 UTC m=+3444.698347531" Nov 03 18:15:02.244093 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.243865 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podStartSLOduration=2.243856515 podStartE2EDuration="2.243856515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.107893457 +0000 UTC m=+3444.562582482" watchObservedRunningTime="2026-11-03 18:15:02.243856515 +0000 UTC m=+3444.698545541" Nov 03 18:15:02.674195 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.674158 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:02.680895 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.680704 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:02.680895 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.680760 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:02.730770 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.730543 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:02.730770 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.730668 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:02.730770 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.730709 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:02.872513 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.872461 4885 generic.go:356] "Generic (PLEG): container finished" podID="010173d7-e198-4520-bd04-c779b36e1791" containerID="e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b" exitCode=60 Nov 03 18:15:02.872979 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.872542 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerDied","Data":"e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b"} Nov 03 18:15:02.872979 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.872612 4885 scope.go:122] "RemoveContainer" containerID="d7301031b0366c80443ac5e80ac19f521192150e2414ef7f39524a37fd8f4354" Nov 03 18:15:02.872979 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:02.872848 4885 scope.go:122] "RemoveContainer" containerID="e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b" Nov 03 18:15:02.873197 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:15:02.873040 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4_odh-ai-gateway-infra(010173d7-e198-4520-bd04-c779b36e1791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" podUID="010173d7-e198-4520-bd04-c779b36e1791" Nov 03 18:15:03.674151 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:03.674126 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:03.674385 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:03.674364 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:03.713458 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:03.713425 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:03.713706 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:03.713675 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:03.880238 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:03.879318 4885 scope.go:122] "RemoveContainer" containerID="e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b" Nov 03 18:15:03.880238 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:15:03.879483 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4_odh-ai-gateway-infra(010173d7-e198-4520-bd04-c779b36e1791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" podUID="010173d7-e198-4520-bd04-c779b36e1791" Nov 03 18:15:04.674626 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:04.674593 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:04.674820 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:04.674646 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:04.714094 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:04.713463 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:04.714094 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:04.713521 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:05.390002 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:05.389959 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:15:05.674352 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:05.674264 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:05.674352 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:05.674327 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:05.715611 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:05.715577 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:05.715879 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:05.715843 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:06.674950 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:06.674914 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:06.675362 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:06.674970 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:06.723497 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:06.723460 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:06.723757 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:06.723729 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:07.674568 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:07.674526 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:07.674742 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:07.674596 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:07.712221 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:07.712188 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:07.712696 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:07.712240 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:08.674513 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:08.674470 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 03 18:15:08.674685 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:08.674546 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 03 18:15:08.711464 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:08.711431 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 18:15:08.711661 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:08.711482 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 18:15:10.675455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.675413 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-td42d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:15:10.675891 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.675473 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:15:10.713329 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.713288 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:10.713875 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.713832 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8nps7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:15:10.714061 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.713891 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:15:10.714602 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.714396 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:10.715582 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.715560 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:15:10.785960 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.785912 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:10.786354 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.786335 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:10.787542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:10.787519 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:15:14.655737 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:14.655706 4885 scope.go:122] "RemoveContainer" containerID="e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b" Nov 03 18:15:14.935018 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:14.934987 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerStarted","Data":"e1f0e4e811e1ca22c412fe51decf3f88f57bb55e97c681cfbceebfd3e7fb30d3"} Nov 03 18:15:15.055580 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:15.055533 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" podStartSLOduration=15.055521776 podStartE2EDuration="15.055521776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:15.053447525 +0000 UTC m=+3457.508136547" watchObservedRunningTime="2026-11-03 18:15:15.055521776 +0000 UTC m=+3457.510210800" Nov 03 18:15:15.940464 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:15.940429 4885 generic.go:356] "Generic (PLEG): container finished" podID="010173d7-e198-4520-bd04-c779b36e1791" containerID="e1f0e4e811e1ca22c412fe51decf3f88f57bb55e97c681cfbceebfd3e7fb30d3" exitCode=60 Nov 03 18:15:15.940910 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:15.940502 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerDied","Data":"e1f0e4e811e1ca22c412fe51decf3f88f57bb55e97c681cfbceebfd3e7fb30d3"} Nov 03 18:15:15.940910 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:15.940576 4885 scope.go:122] "RemoveContainer" containerID="e4d730e43e6250337de2a230e5afbbc079f68f011b93675b6a447c403287a50b" Nov 03 18:15:15.941002 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:15.940924 4885 scope.go:122] "RemoveContainer" containerID="e1f0e4e811e1ca22c412fe51decf3f88f57bb55e97c681cfbceebfd3e7fb30d3" Nov 03 18:15:15.941248 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:15:15.941216 4885 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4_odh-ai-gateway-infra(010173d7-e198-4520-bd04-c779b36e1791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" podUID="010173d7-e198-4520-bd04-c779b36e1791" Nov 03 18:15:16.063858 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:16.063820 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:17.096712 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.096684 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:17.261871 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.261794 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\" (UniqueName: \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\") pod \"010173d7-e198-4520-bd04-c779b36e1791\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " Nov 03 18:15:17.262021 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.261899 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\") pod \"010173d7-e198-4520-bd04-c779b36e1791\" (UID: \"010173d7-e198-4520-bd04-c779b36e1791\") " Nov 03 18:15:17.262349 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.262325 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca" pod "010173d7-e198-4520-bd04-c779b36e1791" (UID: "010173d7-e198-4520-bd04-c779b36e1791"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:17.264152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.264120 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc" pod "010173d7-e198-4520-bd04-c779b36e1791" (UID: "010173d7-e198-4520-bd04-c779b36e1791"). InnerVolumeSpecName "kube-api-access-qm7xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:17.362847 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.362814 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm7xc\" (UniqueName: \"kubernetes.io/projected/010173d7-e198-4520-bd04-c779b36e1791-kube-api-access-qm7xc\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:17.362847 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.362842 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/010173d7-e198-4520-bd04-c779b36e1791-maas-api-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:17.951807 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.951730 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" event={"ID":"010173d7-e198-4520-bd04-c779b36e1791","Type":"ContainerDied","Data":"98862db4dcb99a537700f783e75721934bd9d7331f672834f3fc0dba6d0ee085"} Nov 03 18:15:17.951807 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.951784 4885 scope.go:122] "RemoveContainer" containerID="e1f0e4e811e1ca22c412fe51decf3f88f57bb55e97c681cfbceebfd3e7fb30d3" Nov 03 18:15:17.952043 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:17.951738 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4" Nov 03 18:15:18.119355 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:18.119320 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:18.123605 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:18.123582 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-8c9b0f-29895495-xrjs4"] Nov 03 18:15:18.604642 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:15:18.604609 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:15:19.660659 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:19.660631 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010173d7-e198-4520-bd04-c779b36e1791" path="/var/lib/kubelet/pods/010173d7-e198-4520-bd04-c779b36e1791/volumes" Nov 03 18:15:24.068289 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:24.068261 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:15:38.031367 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:38.031333 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:15:47.931151 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:47.931116 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:15:48.608434 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:15:48.608401 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:15:52.414119 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414084 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt"] Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414419 4885 cpu_manager.go:414] "containerMap: removing container" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414430 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414444 4885 cpu_manager.go:414] "containerMap: removing container" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414449 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414455 4885 cpu_manager.go:414] "containerMap: removing container" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414460 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414519 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.414525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414527 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.415027 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.414533 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="010173d7-e198-4520-bd04-c779b36e1791" containerName="cleanup" Nov 03 18:15:52.417332 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.417316 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.472325 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.472291 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt"] Nov 03 18:15:52.521421 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.521390 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:15:52.521900 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.521883 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:15:52.521959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.521883 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 18:15:52.522260 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.522246 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:15:52.599054 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.599023 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkzx\" (UniqueName: \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.599239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.599092 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.599239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.599203 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.700595 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.700509 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.700595 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.700571 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.700780 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.700625 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkzx\" (UniqueName: \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.701167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.701144 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.703123 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.703087 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:52.763609 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:52.763583 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkzx\" (UniqueName: \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\") pod \"maas-api-revoke-sub-f2902d01-4jzxt\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:53.029550 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:53.029464 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:53.400171 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:15:53.400137 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b621622_299d_4770_bde2_4dde35677658.slice/crio-eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713 WatchSource:0}: Error finding container eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713: Status 404 returned error can't find the container with id eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713 Nov 03 18:15:53.402401 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:53.402380 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt"] Nov 03 18:15:54.124925 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:54.124887 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" event={"ID":"7b621622-299d-4770-bde2-4dde35677658","Type":"ContainerStarted","Data":"3c0dc56028ce5157a3761c700f4a26d3f5c59f9e1cec009be0592161ff73fe3e"} Nov 03 18:15:54.125436 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:54.124942 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" event={"ID":"7b621622-299d-4770-bde2-4dde35677658","Type":"ContainerStarted","Data":"eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713"} Nov 03 18:15:54.244965 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:54.244905 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" podStartSLOduration=2.244892574 podStartE2EDuration="2.244892574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:54.243508904 +0000 UTC m=+3496.698197930" watchObservedRunningTime="2026-11-03 18:15:54.244892574 +0000 UTC m=+3496.699581598" Nov 03 18:15:55.131637 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:55.131602 4885 generic.go:356] "Generic (PLEG): container finished" podID="7b621622-299d-4770-bde2-4dde35677658" containerID="3c0dc56028ce5157a3761c700f4a26d3f5c59f9e1cec009be0592161ff73fe3e" exitCode=0 Nov 03 18:15:55.132036 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:55.131661 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" event={"ID":"7b621622-299d-4770-bde2-4dde35677658","Type":"ContainerDied","Data":"3c0dc56028ce5157a3761c700f4a26d3f5c59f9e1cec009be0592161ff73fe3e"} Nov 03 18:15:56.278160 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.278132 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:56.331239 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.331206 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\" (UniqueName: \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\") pod \"7b621622-299d-4770-bde2-4dde35677658\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " Nov 03 18:15:56.331414 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.331261 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\") pod \"7b621622-299d-4770-bde2-4dde35677658\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " Nov 03 18:15:56.331414 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.331297 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\") pod \"7b621622-299d-4770-bde2-4dde35677658\" (UID: \"7b621622-299d-4770-bde2-4dde35677658\") " Nov 03 18:15:56.331688 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.331661 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca" pod "7b621622-299d-4770-bde2-4dde35677658" (UID: "7b621622-299d-4770-bde2-4dde35677658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:56.333507 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.333479 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx" pod "7b621622-299d-4770-bde2-4dde35677658" (UID: "7b621622-299d-4770-bde2-4dde35677658"). InnerVolumeSpecName "kube-api-access-slkzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:56.333625 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.333556 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca" pod "7b621622-299d-4770-bde2-4dde35677658" (UID: "7b621622-299d-4770-bde2-4dde35677658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:56.432058 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.431976 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b621622-299d-4770-bde2-4dde35677658-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:56.432058 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.432016 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b621622-299d-4770-bde2-4dde35677658-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:56.432058 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:56.432032 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkzx\" (UniqueName: \"kubernetes.io/projected/7b621622-299d-4770-bde2-4dde35677658-kube-api-access-slkzx\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:57.141808 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:57.141777 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" Nov 03 18:15:57.141980 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:57.141776 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt" event={"ID":"7b621622-299d-4770-bde2-4dde35677658","Type":"ContainerDied","Data":"eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713"} Nov 03 18:15:57.141980 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:15:57.141911 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba65e682ab28cd8c35d42895773b1f5b33dd20fdc165fb40e831e02fbf77713" Nov 03 18:16:05.563480 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.563446 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg"] Nov 03 18:16:05.563949 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.563821 4885 cpu_manager.go:414] "containerMap: removing container" podUID="7b621622-299d-4770-bde2-4dde35677658" containerName="revoke-keys" Nov 03 18:16:05.563949 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.563832 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b621622-299d-4770-bde2-4dde35677658" containerName="revoke-keys" Nov 03 18:16:05.563949 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.563886 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b621622-299d-4770-bde2-4dde35677658" containerName="revoke-keys" Nov 03 18:16:05.567145 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.567130 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.620713 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.620685 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg"] Nov 03 18:16:05.621469 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.621444 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:16:05.637904 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.637873 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.638047 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.637934 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.638171 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.638060 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877dt\" (UniqueName: \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.670612 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.670578 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:16:05.670763 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.670627 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:16:05.670763 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.670668 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kf45c\"" Nov 03 18:16:05.738988 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.738952 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.739206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.738996 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877dt\" (UniqueName: \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.739206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.739089 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.739751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.739726 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.741501 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.741482 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.801895 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.801866 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877dt\" (UniqueName: \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\") pod \"maas-api-revoke-sub-6efdf356-2mnvg\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:05.879073 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:05.879044 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:06.245083 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:16:06.245036 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46157d66_e6fa_4e7c_a34b_0e1f4b3eb203.slice/crio-e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870 WatchSource:0}: Error finding container e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870: Status 404 returned error can't find the container with id e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870 Nov 03 18:16:06.246987 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:06.246939 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg"] Nov 03 18:16:07.187216 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:07.187182 4885 generic.go:356] "Generic (PLEG): container finished" podID="46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" containerID="1a067c8c328fd4dec5d99762857bf3927e339417be9bbc2b4f247f111174a19e" exitCode=0 Nov 03 18:16:07.187634 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:07.187222 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" event={"ID":"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203","Type":"ContainerDied","Data":"1a067c8c328fd4dec5d99762857bf3927e339417be9bbc2b4f247f111174a19e"} Nov 03 18:16:07.187634 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:07.187287 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" event={"ID":"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203","Type":"ContainerStarted","Data":"e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870"} Nov 03 18:16:07.366108 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:07.366001 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:16:08.323667 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.323643 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:08.363805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.363771 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\") pod \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " Nov 03 18:16:08.363982 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.363826 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\" (UniqueName: \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\") pod \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " Nov 03 18:16:08.363982 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.363919 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\") pod \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\" (UID: \"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203\") " Nov 03 18:16:08.364187 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.364164 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca" pod "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" (UID: "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:16:08.366189 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.366161 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca" pod "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" (UID: "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:16:08.366317 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.366288 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt" pod "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" (UID: "46157d66-e6fa-4e7c-a34b-0e1f4b3eb203"). InnerVolumeSpecName "kube-api-access-877dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:08.464927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.464841 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:08.464927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.464869 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:08.464927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:08.464878 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-877dt\" (UniqueName: \"kubernetes.io/projected/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203-kube-api-access-877dt\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:09.197325 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:09.197294 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" Nov 03 18:16:09.197325 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:09.197315 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg" event={"ID":"46157d66-e6fa-4e7c-a34b-0e1f4b3eb203","Type":"ContainerDied","Data":"e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870"} Nov 03 18:16:09.197575 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:09.197357 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6f9d4aee9bfeb33e31ca415eb53dca45b0f58b29796f7d741d3f7797ecdf870" Nov 03 18:16:18.604027 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:16:18.603990 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:16:20.836845 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:20.836816 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:16:37.651248 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:16:37.651211 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:16:48.605121 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:16:48.605083 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:17:02.870002 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:17:02.869948 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:17:18.605957 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:17:18.605921 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:17:25.430411 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:17:25.430374 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:17:35.776610 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:17:35.776564 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:17:48.605905 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:17:48.605860 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:17:56.030783 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:17:56.030751 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:18:08.121898 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:18:08.121818 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:18:18.605175 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:18:18.605130 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:18:48.605290 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:18:48.605248 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:19:18.605081 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:19:18.605028 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:19:48.605275 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:19:48.605237 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:20:18.604891 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:20:18.604854 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:20:31.571543 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:31.571505 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:20:31.573782 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:31.571708 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" containerID="cri-o://743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c" gracePeriod=30 Nov 03 18:20:36.813962 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.813940 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:20:36.913951 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.913920 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.913965 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.913992 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914011 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914043 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914111 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914158 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914202 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914241 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\") pod \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\" (UID: \"aba3bb0a-9601-4f3e-bd88-67ab746c9258\") " Nov 03 18:20:36.914495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914307 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:36.914495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914373 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914504 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914530 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914640 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914657 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914671 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:36.914716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914684 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:36.915201 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.914985 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:36.916407 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.916379 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "kube-api-access-z2cmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:36.916532 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.916432 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:36.916532 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.916436 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:36.916532 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:36.916453 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo" pod "aba3bb0a-9601-4f3e-bd88-67ab746c9258" (UID: "aba3bb0a-9601-4f3e-bd88-67ab746c9258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:20:37.015330 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.015300 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:37.015330 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.015327 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:37.015330 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.015336 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:37.015559 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.015345 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2cmc\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-kube-api-access-z2cmc\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:37.015559 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.015355 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba3bb0a-9601-4f3e-bd88-67ab746c9258-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:37.371591 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.371553 4885 generic.go:356] "Generic (PLEG): container finished" podID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerID="743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c" exitCode=0 Nov 03 18:20:37.371767 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.371622 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" Nov 03 18:20:37.371767 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.371650 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" event={"ID":"aba3bb0a-9601-4f3e-bd88-67ab746c9258","Type":"ContainerDied","Data":"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c"} Nov 03 18:20:37.371767 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.371745 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d" event={"ID":"aba3bb0a-9601-4f3e-bd88-67ab746c9258","Type":"ContainerDied","Data":"dd97414b8b269548d5167d17f58df85613f7b089c634557eb0a59ef4725be08d"} Nov 03 18:20:37.371929 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.371776 4885 scope.go:122] "RemoveContainer" containerID="743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c" Nov 03 18:20:37.380892 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.380870 4885 scope.go:122] "RemoveContainer" containerID="743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c" Nov 03 18:20:37.381189 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:20:37.381169 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c\": container with ID starting with 743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c not found: ID does not exist" containerID="743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c" Nov 03 18:20:37.381250 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.381198 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c"} err="failed to get container status \"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c\": rpc error: code = NotFound desc = could not find container \"743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c\": container with ID starting with 743fb3ae831438298561223347f641b9961e366709813e2b426ca45c1fcb148c not found: ID does not exist" Nov 03 18:20:37.570183 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.570149 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:20:37.621014 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.620980 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-td42d"] Nov 03 18:20:37.660504 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:37.660429 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" path="/var/lib/kubelet/pods/aba3bb0a-9601-4f3e-bd88-67ab746c9258/volumes" Nov 03 18:20:48.605439 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:20:48.605391 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:20:57.094131 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:57.094095 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt"] Nov 03 18:20:57.144825 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:57.144791 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2902d01-4jzxt"] Nov 03 18:20:57.660756 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:20:57.660723 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b621622-299d-4770-bde2-4dde35677658" path="/var/lib/kubelet/pods/7b621622-299d-4770-bde2-4dde35677658/volumes" Nov 03 18:21:09.063892 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:21:09.063814 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg"] Nov 03 18:21:09.114933 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:21:09.114905 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6efdf356-2mnvg"] Nov 03 18:21:09.659982 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:21:09.659953 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" path="/var/lib/kubelet/pods/46157d66-e6fa-4e7c-a34b-0e1f4b3eb203/volumes" Nov 03 18:21:18.604406 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:21:18.604372 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:21:25.110508 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:21:25.110471 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:21:47.157249 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:21:47.157210 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:21:48.605677 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:21:48.605638 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:22:02.288234 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:02.288200 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:22:02.288780 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:02.288392 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" containerID="cri-o://83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae" gracePeriod=30 Nov 03 18:22:07.325119 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:22:07.325057 4885 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f32ece_3f23_4c8c_85b3_ec6f95e92019.slice/crio-83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f32ece_3f23_4c8c_85b3_ec6f95e92019.slice/crio-conmon-83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae.scope\": RecentStats: unable to find data in memory cache]" Nov 03 18:22:07.535969 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.535945 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:22:07.629291 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629258 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629305 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629329 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629386 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629412 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629453 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629712 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629475 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629712 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629521 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629712 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629551 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\") pod \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\" (UID: \"03f32ece-3f23-4c8c-85b3-ec6f95e92019\") " Nov 03 18:22:07.629883 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629840 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:22:07.629883 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.629844 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:07.630251 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.630052 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:07.630251 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.630092 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:07.630251 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.630105 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:07.631805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.631776 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:07.631805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.631794 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "kube-api-access-rt6bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:07.632169 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.632145 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:07.632169 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.632159 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo" pod "03f32ece-3f23-4c8c-85b3-ec6f95e92019" (UID: "03f32ece-3f23-4c8c-85b3-ec6f95e92019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:22:07.730222 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730186 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730222 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730214 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730222 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730223 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730233 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730241 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730249 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730258 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt6bt\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-kube-api-access-rt6bt\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730268 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.730454 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.730276 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03f32ece-3f23-4c8c-85b3-ec6f95e92019-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:22:07.752748 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.752713 4885 generic.go:356] "Generic (PLEG): container finished" podID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerID="83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae" exitCode=0 Nov 03 18:22:07.752900 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.752764 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" Nov 03 18:22:07.752900 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.752766 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" event={"ID":"03f32ece-3f23-4c8c-85b3-ec6f95e92019","Type":"ContainerDied","Data":"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae"} Nov 03 18:22:07.752900 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.752881 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7" event={"ID":"03f32ece-3f23-4c8c-85b3-ec6f95e92019","Type":"ContainerDied","Data":"d0f21222b790ac057f5dd0ff41962d6b3ce3b6add78fb69925087ef54c6c44c9"} Nov 03 18:22:07.753010 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.752902 4885 scope.go:122] "RemoveContainer" containerID="83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae" Nov 03 18:22:07.761168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.761147 4885 scope.go:122] "RemoveContainer" containerID="83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae" Nov 03 18:22:07.761432 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:22:07.761413 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae\": container with ID starting with 83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae not found: ID does not exist" containerID="83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae" Nov 03 18:22:07.761492 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.761442 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae"} err="failed to get container status \"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae\": rpc error: code = NotFound desc = could not find container \"83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae\": container with ID starting with 83709b9d9166f76fbf6df961eb3447a19c3426e4dad4a834ff52d557688dc9ae not found: ID does not exist" Nov 03 18:22:07.924319 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.924235 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:22:07.928333 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:07.928304 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8nps7"] Nov 03 18:22:09.661407 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:09.661368 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" path="/var/lib/kubelet/pods/03f32ece-3f23-4c8c-85b3-ec6f95e92019/volumes" Nov 03 18:22:09.668908 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:09.668872 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:22:18.606405 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:22:18.606368 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:22:22.945354 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:22.945312 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:22:38.782677 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:38.782601 4885 scope.go:122] "RemoveContainer" containerID="1a067c8c328fd4dec5d99762857bf3927e339417be9bbc2b4f247f111174a19e" Nov 03 18:22:38.790472 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:22:38.790455 4885 scope.go:122] "RemoveContainer" containerID="3c0dc56028ce5157a3761c700f4a26d3f5c59f9e1cec009be0592161ff73fe3e" Nov 03 18:22:48.605294 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:22:48.605258 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:23:18.605791 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:23:18.605757 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:23:48.605844 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:23:48.605804 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:24:18.605521 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:24:18.605433 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:24:48.606293 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:24:48.606243 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:25:18.606040 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:25:18.605998 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:25:45.592119 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:25:45.592083 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:25:48.608333 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:25:48.608287 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:25:58.854157 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:25:58.854121 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:26:13.926547 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:13.926516 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:26:18.604352 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:26:18.604314 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:26:24.132359 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:24.132328 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:26:37.589596 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:37.589565 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:26:48.603907 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:26:48.603869 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:26:49.467193 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:49.467162 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:26:49.467414 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:49.467391 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" containerID="cri-o://b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff" gracePeriod=30 Nov 03 18:26:51.028564 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:51.028530 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:26:54.707469 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.707439 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:26:54.826618 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826527 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826618 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826576 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826618 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826597 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826650 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826678 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826711 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826738 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826786 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.826901 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826810 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\") pod \"67af5e29-dc79-419b-9e03-2fb871b2de27\" (UID: \"67af5e29-dc79-419b-9e03-2fb871b2de27\") " Nov 03 18:26:54.827279 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.826972 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:26:54.827279 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.827128 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:26:54.827279 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.827137 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:26:54.827445 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.827297 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:26:54.827445 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.827402 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:26:54.829348 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.829319 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "kube-api-access-xq5wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:26:54.829525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.829507 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:26:54.829620 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.829570 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:26:54.829712 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.829689 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy" pod "67af5e29-dc79-419b-9e03-2fb871b2de27" (UID: "67af5e29-dc79-419b-9e03-2fb871b2de27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:26:54.927616 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927582 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5wx\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-kube-api-access-xq5wx\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927616 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927613 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927616 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927622 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927616 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927632 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927645 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927654 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927662 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67af5e29-dc79-419b-9e03-2fb871b2de27-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927670 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67af5e29-dc79-419b-9e03-2fb871b2de27-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.927919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.927678 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67af5e29-dc79-419b-9e03-2fb871b2de27-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:26:54.970342 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.970307 4885 generic.go:356] "Generic (PLEG): container finished" podID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerID="b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff" exitCode=0 Nov 03 18:26:54.970342 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.970349 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" Nov 03 18:26:54.970615 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.970393 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" event={"ID":"67af5e29-dc79-419b-9e03-2fb871b2de27","Type":"ContainerDied","Data":"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff"} Nov 03 18:26:54.970615 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.970458 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9" event={"ID":"67af5e29-dc79-419b-9e03-2fb871b2de27","Type":"ContainerDied","Data":"8e2773aea57e94632e5b3cb4c62bed81f65eed126276d57010cc10ca75ec207e"} Nov 03 18:26:54.970615 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.970489 4885 scope.go:122] "RemoveContainer" containerID="b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff" Nov 03 18:26:54.980143 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.980128 4885 scope.go:122] "RemoveContainer" containerID="b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff" Nov 03 18:26:54.980368 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:26:54.980351 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff\": container with ID starting with b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff not found: ID does not exist" containerID="b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff" Nov 03 18:26:54.980407 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:54.980377 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff"} err="failed to get container status \"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff\": rpc error: code = NotFound desc = could not find container \"b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff\": container with ID starting with b936544c536867cf2f4d2ba63bd1234474989e869ea88d3f659315cb90627aff not found: ID does not exist" Nov 03 18:26:55.138523 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:55.138488 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:26:55.142435 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:55.142410 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qqzc9"] Nov 03 18:26:55.661196 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:26:55.661158 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" path="/var/lib/kubelet/pods/67af5e29-dc79-419b-9e03-2fb871b2de27/volumes" Nov 03 18:27:03.074430 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:03.074350 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:27:18.607389 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:27:18.607349 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:27:20.115424 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:20.115387 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:27:41.138466 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:41.138428 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:27:42.352690 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:42.352660 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 18:27:42.353101 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:42.352862 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" containerID="cri-o://94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df" gracePeriod=30 Nov 03 18:27:44.213307 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:44.213248 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4lcxm_b9b78432-2478-417e-9226-2d987a705dc5/manager/0.log" Nov 03 18:27:44.513462 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:44.513378 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xbdkc_7e51fea8-9d06-422e-9176-70c40477d2ab/manager/0.log" Nov 03 18:27:45.966372 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:45.966336 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wdgfb_a25335e8-e46d-4757-9cf4-0b36465d0b8e/manager/0.log" Nov 03 18:27:46.202882 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:46.202850 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sdh8h_2003487e-5c5d-4661-bfdb-b9b72e729cb6/limitador/0.log" Nov 03 18:27:47.597431 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.597406 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 18:27:47.673290 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673263 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673332 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673356 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673380 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673400 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673456 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673424 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673475 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673503 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673526 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673534 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\") pod \"9936961a-214f-4d66-b54f-6f2ce6b15376\" (UID: \"9936961a-214f-4d66-b54f-6f2ce6b15376\") " Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673684 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:27:47.673785 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673704 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673828 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673951 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673970 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.673986 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.674004 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.674107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.674010 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:27:47.675777 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.675752 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:27:47.675777 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.675769 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:27:47.675926 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.675812 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:27:47.675926 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.675840 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz" pod "9936961a-214f-4d66-b54f-6f2ce6b15376" (UID: "9936961a-214f-4d66-b54f-6f2ce6b15376"). InnerVolumeSpecName "kube-api-access-rbzcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:27:47.774954 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.774929 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.774954 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.774952 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9936961a-214f-4d66-b54f-6f2ce6b15376-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.774954 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.774961 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzcz\" (UniqueName: \"kubernetes.io/projected/9936961a-214f-4d66-b54f-6f2ce6b15376-kube-api-access-rbzcz\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.775206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.774971 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.775206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.774979 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9936961a-214f-4d66-b54f-6f2ce6b15376-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:27:47.959364 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:47.959287 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-r7rw5_19e0a745-46bf-4cb6-a26b-bc8c643aa60d/manager/0.log" Nov 03 18:27:48.186655 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.186624 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xj552_6b2b23ec-041e-4b09-bc89-fa5e2399b5fc/manager/0.log" Nov 03 18:27:48.194501 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.194476 4885 generic.go:356] "Generic (PLEG): container finished" podID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerID="94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df" exitCode=0 Nov 03 18:27:48.194632 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.194526 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" Nov 03 18:27:48.194632 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.194555 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" event={"ID":"9936961a-214f-4d66-b54f-6f2ce6b15376","Type":"ContainerDied","Data":"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df"} Nov 03 18:27:48.194632 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.194601 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9" event={"ID":"9936961a-214f-4d66-b54f-6f2ce6b15376","Type":"ContainerDied","Data":"309c27e17ed404d0027a28ff79ab7a746b9372f012f7be376d497bf01eb08997"} Nov 03 18:27:48.194632 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.194622 4885 scope.go:122] "RemoveContainer" containerID="94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df" Nov 03 18:27:48.203908 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.203891 4885 scope.go:122] "RemoveContainer" containerID="94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df" Nov 03 18:27:48.204204 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:27:48.204187 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df\": container with ID starting with 94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df not found: ID does not exist" containerID="94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df" Nov 03 18:27:48.204284 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.204215 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df"} err="failed to get container status \"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df\": rpc error: code = NotFound desc = could not find container \"94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df\": container with ID starting with 94f2775fafa13e0d5ae46966dcca10094e130efb6c4b07801fe017b818d0a6df not found: ID does not exist" Nov 03 18:27:48.362751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.362658 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 18:27:48.413945 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.413909 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-8c9b0f-openshift-default-5cf4b7d76f-pknl9"] Nov 03 18:27:48.437962 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.437932 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hqg4v_7fbbf0df-7541-4486-bc3c-861bea91ef4e/manager/0.log" Nov 03 18:27:48.606297 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:27:48.606264 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:27:48.912093 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:48.912045 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c4dd589d-rbr8b_ee922e87-96e2-466a-baf9-082bc2238f2c/manager/0.log" Nov 03 18:27:49.661002 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:27:49.660966 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" path="/var/lib/kubelet/pods/9936961a-214f-4d66-b54f-6f2ce6b15376/volumes" Nov 03 18:28:12.170739 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:28:12.170699 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:28:18.606358 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:28:18.606317 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:28:48.605011 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:28:48.604969 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:28:59.118611 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:28:59.118570 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:29:00.429107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.429054 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 18:29:00.429540 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.429277 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" podUID="ee922e87-96e2-466a-baf9-082bc2238f2c" containerName="manager" containerID="cri-o://d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3" gracePeriod=30 Nov 03 18:29:00.683568 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.683496 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 18:29:00.791223 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.791186 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\") pod \"ee922e87-96e2-466a-baf9-082bc2238f2c\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " Nov 03 18:29:00.791394 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.791237 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\" (UniqueName: \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\") pod \"ee922e87-96e2-466a-baf9-082bc2238f2c\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " Nov 03 18:29:00.791394 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.791259 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\") pod \"ee922e87-96e2-466a-baf9-082bc2238f2c\" (UID: \"ee922e87-96e2-466a-baf9-082bc2238f2c\") " Nov 03 18:29:00.793618 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.793583 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs" pod "ee922e87-96e2-466a-baf9-082bc2238f2c" (UID: "ee922e87-96e2-466a-baf9-082bc2238f2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:29:00.793618 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.793603 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert" pod "ee922e87-96e2-466a-baf9-082bc2238f2c" (UID: "ee922e87-96e2-466a-baf9-082bc2238f2c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:29:00.793770 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.793671 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h" pod "ee922e87-96e2-466a-baf9-082bc2238f2c" (UID: "ee922e87-96e2-466a-baf9-082bc2238f2c"). InnerVolumeSpecName "kube-api-access-fxf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:29:00.892291 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.892261 4885 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-webhook-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:29:00.892291 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.892289 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxf9h\" (UniqueName: \"kubernetes.io/projected/ee922e87-96e2-466a-baf9-082bc2238f2c-kube-api-access-fxf9h\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:29:00.892291 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:00.892301 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee922e87-96e2-466a-baf9-082bc2238f2c-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:29:01.491220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.491183 4885 generic.go:356] "Generic (PLEG): container finished" podID="ee922e87-96e2-466a-baf9-082bc2238f2c" containerID="d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3" exitCode=0 Nov 03 18:29:01.491721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.491234 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" Nov 03 18:29:01.491721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.491256 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" event={"ID":"ee922e87-96e2-466a-baf9-082bc2238f2c","Type":"ContainerDied","Data":"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3"} Nov 03 18:29:01.491721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.491298 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c4dd589d-rbr8b" event={"ID":"ee922e87-96e2-466a-baf9-082bc2238f2c","Type":"ContainerDied","Data":"ffb68ac9056f552b261cc867415508f32fd60aaae738a1e7bf59a593144cfc91"} Nov 03 18:29:01.491721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.491320 4885 scope.go:122] "RemoveContainer" containerID="d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3" Nov 03 18:29:01.500994 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.500974 4885 scope.go:122] "RemoveContainer" containerID="d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3" Nov 03 18:29:01.501274 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:29:01.501255 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3\": container with ID starting with d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3 not found: ID does not exist" containerID="d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3" Nov 03 18:29:01.501342 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.501287 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3"} err="failed to get container status \"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3\": rpc error: code = NotFound desc = could not find container \"d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3\": container with ID starting with d373b7233a1180b61389567612be782dd6d9d29ab7dacac4aab4b391aefbdeb3 not found: ID does not exist" Nov 03 18:29:01.671516 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.671479 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 18:29:01.682295 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:01.682267 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75c4dd589d-rbr8b"] Nov 03 18:29:03.661103 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:03.661050 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee922e87-96e2-466a-baf9-082bc2238f2c" path="/var/lib/kubelet/pods/ee922e87-96e2-466a-baf9-082bc2238f2c/volumes" Nov 03 18:29:18.605963 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:29:18.605926 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:29:35.777546 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777509 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777806 4885 cpu_manager.go:414] "containerMap: removing container" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777815 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777827 4885 cpu_manager.go:414] "containerMap: removing container" podUID="ee922e87-96e2-466a-baf9-082bc2238f2c" containerName="manager" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777832 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee922e87-96e2-466a-baf9-082bc2238f2c" containerName="manager" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777840 4885 cpu_manager.go:414] "containerMap: removing container" podUID="46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" containerName="revoke-keys" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777845 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" containerName="revoke-keys" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777853 4885 cpu_manager.go:414] "containerMap: removing container" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777859 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777875 4885 cpu_manager.go:414] "containerMap: removing container" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777880 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777886 4885 cpu_manager.go:414] "containerMap: removing container" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777891 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777940 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee922e87-96e2-466a-baf9-082bc2238f2c" containerName="manager" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777946 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="46157d66-e6fa-4e7c-a34b-0e1f4b3eb203" containerName="revoke-keys" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777953 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="67af5e29-dc79-419b-9e03-2fb871b2de27" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777961 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9936961a-214f-4d66-b54f-6f2ce6b15376" containerName="istio-proxy" Nov 03 18:29:35.777959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777966 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="03f32ece-3f23-4c8c-85b3-ec6f95e92019" containerName="istio-proxy" Nov 03 18:29:35.778501 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.777973 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba3bb0a-9601-4f3e-bd88-67ab746c9258" containerName="istio-proxy" Nov 03 18:29:35.780764 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.780748 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.784913 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.784888 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785044 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.784934 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785044 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.784960 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785160 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785136 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785197 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785178 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785250 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785234 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785320 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785305 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785361 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785329 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.785361 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.785346 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzp5z\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.834884 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.834857 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\"" Nov 03 18:29:35.885720 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885687 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.885720 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885726 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.885972 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885742 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.885972 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885769 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.885972 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885906 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.885972 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885957 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.885997 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886024 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886050 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzp5z\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886169 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886220 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886209 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886489 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886432 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886619 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886595 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.886709 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.886663 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.888375 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.888353 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.888814 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.888798 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.948534 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.948500 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.948714 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.948588 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzp5z\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\") pod \"maas-default-gateway-openshift-default-58c6585447-927sl\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:35.981831 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:35.981801 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 18:29:36.093135 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.093024 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:36.511487 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:29:36.511458 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9795628_59b5_44cb_baf3_d93ea91ce738.slice/crio-4549bc42b08cbfbd5b4b24169b82722970bd6cd38ef74336b7c366622465eff8 WatchSource:0}: Error finding container 4549bc42b08cbfbd5b4b24169b82722970bd6cd38ef74336b7c366622465eff8: Status 404 returned error can't find the container with id 4549bc42b08cbfbd5b4b24169b82722970bd6cd38ef74336b7c366622465eff8 Nov 03 18:29:36.512361 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.512336 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 18:29:36.513573 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.513541 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:36.513671 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.513603 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:36.513671 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.513654 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:36.630677 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.630636 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" event={"ID":"b9795628-59b5-44cb-baf3-d93ea91ce738","Type":"ContainerStarted","Data":"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec"} Nov 03 18:29:36.630877 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.630697 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" event={"ID":"b9795628-59b5-44cb-baf3-d93ea91ce738","Type":"ContainerStarted","Data":"4549bc42b08cbfbd5b4b24169b82722970bd6cd38ef74336b7c366622465eff8"} Nov 03 18:29:36.756020 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:36.755957 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podStartSLOduration=1.755932387 podStartE2EDuration="1.755932387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:36.754776648 +0000 UTC m=+4319.209465674" watchObservedRunningTime="2026-11-03 18:29:36.755932387 +0000 UTC m=+4319.210621412" Nov 03 18:29:37.093471 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:37.093438 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:37.094711 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:37.094691 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:37.094774 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:37.094735 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:38.094430 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:38.094373 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:38.094951 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:38.094460 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:39.094318 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:39.094282 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:39.094496 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:39.094343 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:40.093582 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:40.093540 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:40.093767 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:40.093598 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:41.093967 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:41.093928 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:41.094407 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:41.093981 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:42.094412 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:42.094374 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:42.094818 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:42.094440 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:43.094149 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:43.094109 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:43.094342 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:43.094166 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:44.094280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:44.094240 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 18:29:44.094680 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:44.094318 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 18:29:45.641956 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:45.641898 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:45.642475 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:45.642451 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:45.643165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:45.643145 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 18:29:48.605796 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:29:48.605760 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:29:50.881402 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.881369 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 18:29:50.886030 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.886012 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.936874 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.936843 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937019 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.936883 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937019 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.936904 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937019 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.936980 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937019 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.937009 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937184 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.937034 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937184 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.937100 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937184 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.937163 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bws2\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.937279 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.937197 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:50.990897 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:50.990865 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 18:29:51.037709 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037676 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.037709 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037712 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.037937 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037729 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.037937 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037747 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.037937 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037764 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.037937 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037795 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038181 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.037933 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038181 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038044 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bws2\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038181 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038094 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038343 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038305 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038402 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038359 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038466 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038431 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038528 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038484 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.038528 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.038519 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.040589 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.040555 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.040734 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.040597 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.101854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.101814 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.102015 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.101921 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bws2\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\") pod \"maas-default-gateway-openshift-default-58c6585447-csqjk\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.198590 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.198512 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:51.613825 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:29:51.613779 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a32526_b77b_4ca2_872b_f5c928ddd077.slice/crio-4b90ce8ccc1264ddca833d38606e162f8accf34072f0110356dc7da45236d9ae WatchSource:0}: Error finding container 4b90ce8ccc1264ddca833d38606e162f8accf34072f0110356dc7da45236d9ae: Status 404 returned error can't find the container with id 4b90ce8ccc1264ddca833d38606e162f8accf34072f0110356dc7da45236d9ae Nov 03 18:29:51.615293 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.615253 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 18:29:51.615764 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.615737 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:51.615843 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.615808 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:51.615915 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.615848 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:29:51.695373 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.695340 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" event={"ID":"69a32526-b77b-4ca2-872b-f5c928ddd077","Type":"ContainerStarted","Data":"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734"} Nov 03 18:29:51.695538 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.695390 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" event={"ID":"69a32526-b77b-4ca2-872b-f5c928ddd077","Type":"ContainerStarted","Data":"4b90ce8ccc1264ddca833d38606e162f8accf34072f0110356dc7da45236d9ae"} Nov 03 18:29:51.819701 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:51.819648 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podStartSLOduration=1.819633204 podStartE2EDuration="1.819633204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:51.81842832 +0000 UTC m=+4334.273117345" watchObservedRunningTime="2026-11-03 18:29:51.819633204 +0000 UTC m=+4334.274322227" Nov 03 18:29:52.199247 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:52.199209 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:29:52.200634 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:52.200606 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:52.200750 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:52.200654 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:53.199100 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:53.199044 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:53.199280 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:53.199129 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:54.199852 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:54.199815 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:54.200370 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:54.199891 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:55.199868 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:55.199835 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:55.200290 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:55.199890 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:56.199327 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:56.199294 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:56.199502 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:56.199348 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:57.199710 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:57.199678 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:57.200168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:57.199738 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:58.199269 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:58.199235 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:58.199446 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:58.199303 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:29:59.199572 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:59.199491 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:29:59.199961 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:29:59.199561 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 18:30:00.709462 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:00.709431 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:30:00.709974 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:00.709904 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:30:00.710805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:00.710780 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 18:30:05.024005 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:05.023957 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:30:12.599545 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.599502 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh"] Nov 03 18:30:12.603036 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.603013 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.656875 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.656841 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh"] Nov 03 18:30:12.706955 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.706917 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:30:12.706955 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.706926 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:30:12.707218 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.707129 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:30:12.708145 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.708127 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\"" Nov 03 18:30:12.726161 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.726130 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.726274 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.726171 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.726274 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.726261 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pj2\" (UniqueName: \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.827157 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.827127 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pj2\" (UniqueName: \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.827315 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.827224 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.827315 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.827254 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.828008 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.827945 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.829990 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.829962 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.889833 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.889809 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pj2\" (UniqueName: \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\") pod \"maas-api-revoke-sub-f904bff9-mk8fh\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:12.915637 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:12.915613 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:13.283263 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:30:13.283234 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb96bb5_1005_4c58_8cb5_b0ec3f2790b1.slice/crio-8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09 WatchSource:0}: Error finding container 8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09: Status 404 returned error can't find the container with id 8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09 Nov 03 18:30:13.284518 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:13.284495 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh"] Nov 03 18:30:13.804102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:13.803991 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" event={"ID":"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1","Type":"ContainerStarted","Data":"3b44d20a91e6558d8bbda73efbaebf61b52ef5e024f623bdedca7c4346b47ef9"} Nov 03 18:30:13.804102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:13.804094 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" event={"ID":"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1","Type":"ContainerStarted","Data":"8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09"} Nov 03 18:30:13.974506 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:13.974457 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" podStartSLOduration=1.9744441099999999 podStartE2EDuration="1.97444411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:13.922880731 +0000 UTC m=+4356.377569757" watchObservedRunningTime="2026-11-03 18:30:13.97444411 +0000 UTC m=+4356.429133135" Nov 03 18:30:14.809317 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:14.809280 4885 generic.go:356] "Generic (PLEG): container finished" podID="0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" containerID="3b44d20a91e6558d8bbda73efbaebf61b52ef5e024f623bdedca7c4346b47ef9" exitCode=0 Nov 03 18:30:14.809725 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:14.809369 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" event={"ID":"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1","Type":"ContainerDied","Data":"3b44d20a91e6558d8bbda73efbaebf61b52ef5e024f623bdedca7c4346b47ef9"} Nov 03 18:30:15.948877 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:15.948851 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:16.051304 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.051268 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\") pod \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " Nov 03 18:30:16.051477 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.051309 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\" (UniqueName: \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\") pod \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " Nov 03 18:30:16.051477 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.051377 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\") pod \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\" (UID: \"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1\") " Nov 03 18:30:16.051786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.051764 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca" pod "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" (UID: "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:30:16.053531 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.053506 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca" pod "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" (UID: "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:30:16.053645 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.053618 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2" pod "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" (UID: "0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1"). InnerVolumeSpecName "kube-api-access-f8pj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:30:16.152455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.152416 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:30:16.152455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.152445 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8pj2\" (UniqueName: \"kubernetes.io/projected/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-kube-api-access-f8pj2\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:30:16.152455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.152455 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:30:16.819486 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.819450 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" event={"ID":"0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1","Type":"ContainerDied","Data":"8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09"} Nov 03 18:30:16.819486 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.819478 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh" Nov 03 18:30:16.819701 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:16.819501 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b0cfffb6d1bd222a9d7c3e6a10e68a8f663674949964ea88909165cc6e1ff09" Nov 03 18:30:18.605587 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:30:18.605545 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:30:25.365259 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:25.365222 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:30:34.861635 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:34.861600 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:30:36.257414 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.257383 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 18:30:36.257902 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.257695 4885 cpu_manager.go:414] "containerMap: removing container" podUID="0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" containerName="revoke-keys" Nov 03 18:30:36.257902 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.257705 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" containerName="revoke-keys" Nov 03 18:30:36.257902 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.257762 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" containerName="revoke-keys" Nov 03 18:30:36.260698 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.260680 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.338695 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.338663 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 18:30:36.351898 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.351865 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352095 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.351910 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352095 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352010 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmk7\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352095 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352056 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352217 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352196 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352262 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352243 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352318 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352301 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352413 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352391 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.352449 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.352436 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.452825 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452787 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452841 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452866 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452886 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452905 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452927 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452951 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmk7\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453012 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.452970 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453419 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453021 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453419 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453305 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453419 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453333 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453585 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453466 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453626 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453597 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.453702 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.453684 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.455545 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.455525 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.455673 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.455653 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.515750 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.515670 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.515919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.515767 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmk7\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\") pod \"maas-default-gateway-openshift-default-58c6585447-zg2g2\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.573647 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.573618 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:36.989037 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:30:36.989008 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b428a3_01ca_46e2_a264_3415b03bf9f6.slice/crio-4bb395f5b9139973a816fae620426f7dda77e6c3d5d780fb82ded1877e99d245 WatchSource:0}: Error finding container 4bb395f5b9139973a816fae620426f7dda77e6c3d5d780fb82ded1877e99d245: Status 404 returned error can't find the container with id 4bb395f5b9139973a816fae620426f7dda77e6c3d5d780fb82ded1877e99d245 Nov 03 18:30:36.991124 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.991099 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 18:30:36.991541 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.991511 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:36.991585 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.991574 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:36.991619 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:36.991601 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:37.929782 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:37.929734 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" event={"ID":"90b428a3-01ca-46e2-a264-3415b03bf9f6","Type":"ContainerStarted","Data":"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639"} Nov 03 18:30:37.930360 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:37.929811 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" event={"ID":"90b428a3-01ca-46e2-a264-3415b03bf9f6","Type":"ContainerStarted","Data":"4bb395f5b9139973a816fae620426f7dda77e6c3d5d780fb82ded1877e99d245"} Nov 03 18:30:38.574539 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:38.574505 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:38.575844 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:38.575818 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:38.575964 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:38.575866 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:39.574039 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:39.574003 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:39.574451 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:39.574058 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:40.574681 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:40.574644 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:40.575106 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:40.574709 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:41.574723 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:41.574684 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:41.575162 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:41.574737 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:42.574890 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:42.574789 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:42.577429 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:42.574880 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:43.574571 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:43.574526 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:43.574780 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:43.574585 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:44.574515 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:44.574476 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:44.574912 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:44.574539 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:45.574053 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:45.574016 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 18:30:45.574248 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:45.574079 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 18:30:46.649183 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:46.649130 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podStartSLOduration=10.649117622 podStartE2EDuration="10.649117622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:38.108409128 +0000 UTC m=+4380.563098156" watchObservedRunningTime="2026-11-03 18:30:46.649117622 +0000 UTC m=+4389.103806652" Nov 03 18:30:46.649922 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:46.649907 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:30:47.575113 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:47.575049 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:30:47.575318 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:47.575140 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:30:48.605911 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:30:48.605868 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:30:48.617221 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:48.617191 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:30:48.617387 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:48.617286 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:30:49.660284 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:49.660250 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:30:49.660824 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:49.660303 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:30:49.934425 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:49.934346 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:49.934996 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:49.934974 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:49.935726 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:49.935706 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 18:30:54.954834 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:30:54.954799 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:31:18.606119 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:31:18.606077 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:31:33.518930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:31:33.518847 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:31:43.948848 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:31:43.948810 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:31:48.605566 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:31:48.605525 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:32:18.605552 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:32:18.605515 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:32:24.529817 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:32:24.529782 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:32:48.605120 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:32:48.605060 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:32:50.333675 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:32:50.333633 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:33:03.056854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:03.056771 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:33:18.605415 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:33:18.605372 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:33:21.417570 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.417529 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:33:21.420668 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.420642 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.508917 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.508881 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.509125 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.508936 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.509125 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.509086 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgh8\" (UniqueName: \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.509125 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.509117 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.525721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.525684 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:33:21.525721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.525720 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:33:21.525721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.525733 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:33:21.526041 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.525766 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 18:33:21.526041 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.525774 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-57d07b-serving-cert\"" Nov 03 18:33:21.527366 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.527341 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:33:21.609676 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.609640 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.609867 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.609691 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgh8\" (UniqueName: \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.609867 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.609710 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.609867 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.609759 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.609867 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:33:21.609858 4885 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-serving-cert: secret "maas-api-e2e-worker-single-main-57d07b-serving-cert" not found Nov 03 18:33:21.610038 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:33:21.609933 4885 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls podName:eb1a97a5-e924-4286-a6c6-443f57262f7c nodeName:}" failed. No retries permitted until 2026-11-03 18:33:22.109911394 +0000 UTC m=+4544.564600399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls") pod "maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" (UID: "eb1a97a5-e924-4286-a6c6-443f57262f7c") : secret "maas-api-e2e-worker-single-main-57d07b-serving-cert" not found Nov 03 18:33:21.610533 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.610504 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.612269 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.612248 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:21.672750 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:21.672669 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgh8\" (UniqueName: \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:22.115519 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.115422 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:22.118238 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.118206 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:22.333704 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.333664 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:22.703454 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:33:22.703413 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1a97a5_e924_4286_a6c6_443f57262f7c.slice/crio-086b0ecad69b8c3779c012fbfc3cab932549fc946268b08687d06f9cec54db98 WatchSource:0}: Error finding container 086b0ecad69b8c3779c012fbfc3cab932549fc946268b08687d06f9cec54db98: Status 404 returned error can't find the container with id 086b0ecad69b8c3779c012fbfc3cab932549fc946268b08687d06f9cec54db98 Nov 03 18:33:22.704531 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.704508 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:33:22.705468 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.705450 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:33:22.727680 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:22.727640 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" event={"ID":"eb1a97a5-e924-4286-a6c6-443f57262f7c","Type":"ContainerStarted","Data":"086b0ecad69b8c3779c012fbfc3cab932549fc946268b08687d06f9cec54db98"} Nov 03 18:33:23.733315 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:23.733266 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" event={"ID":"eb1a97a5-e924-4286-a6c6-443f57262f7c","Type":"ContainerStarted","Data":"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033"} Nov 03 18:33:23.733722 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:23.733347 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:23.861036 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:23.860935 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" podStartSLOduration=2.000558125 podStartE2EDuration="2.860921723s" totalImagesPullingTime="860.363598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:33:23.860349911 +0000 UTC m=+4546.315038936" watchObservedRunningTime="2026-11-03 18:33:23.860921723 +0000 UTC m=+4546.315610745" Nov 03 18:33:27.643935 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:27.643896 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:33:29.743869 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:29.743842 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:33:47.536295 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:33:47.536261 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:33:48.607264 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:33:48.607225 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:34:13.650592 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:13.650557 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:34:18.606950 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:34:18.606910 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:34:41.958997 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:41.958959 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:34:48.606484 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:34:48.606443 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:34:50.776562 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:50.776521 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:34:50.777028 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:50.776752 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" podUID="eb1a97a5-e924-4286-a6c6-443f57262f7c" containerName="maas-api" containerID="cri-o://bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033" gracePeriod=30 Nov 03 18:34:52.018652 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.018625 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:34:52.128961 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.128927 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\") pod \"eb1a97a5-e924-4286-a6c6-443f57262f7c\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " Nov 03 18:34:52.129201 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.128968 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\") pod \"eb1a97a5-e924-4286-a6c6-443f57262f7c\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " Nov 03 18:34:52.129201 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129026 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") pod \"eb1a97a5-e924-4286-a6c6-443f57262f7c\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " Nov 03 18:34:52.129201 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129051 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\" (UniqueName: \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\") pod \"eb1a97a5-e924-4286-a6c6-443f57262f7c\" (UID: \"eb1a97a5-e924-4286-a6c6-443f57262f7c\") " Nov 03 18:34:52.129435 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129399 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca" pod "eb1a97a5-e924-4286-a6c6-443f57262f7c" (UID: "eb1a97a5-e924-4286-a6c6-443f57262f7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:34:52.129921 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129891 4885 generic.go:356] "Generic (PLEG): container finished" podID="eb1a97a5-e924-4286-a6c6-443f57262f7c" containerID="bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033" exitCode=0 Nov 03 18:34:52.130037 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129937 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" event={"ID":"eb1a97a5-e924-4286-a6c6-443f57262f7c","Type":"ContainerDied","Data":"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033"} Nov 03 18:34:52.130037 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129999 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" event={"ID":"eb1a97a5-e924-4286-a6c6-443f57262f7c","Type":"ContainerDied","Data":"086b0ecad69b8c3779c012fbfc3cab932549fc946268b08687d06f9cec54db98"} Nov 03 18:34:52.130037 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.130022 4885 scope.go:122] "RemoveContainer" containerID="bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033" Nov 03 18:34:52.130037 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.129947 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf" Nov 03 18:34:52.131721 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.131696 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8" pod "eb1a97a5-e924-4286-a6c6-443f57262f7c" (UID: "eb1a97a5-e924-4286-a6c6-443f57262f7c"). InnerVolumeSpecName "kube-api-access-cbgh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:34:52.131858 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.131838 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls" pod "eb1a97a5-e924-4286-a6c6-443f57262f7c" (UID: "eb1a97a5-e924-4286-a6c6-443f57262f7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:34:52.132873 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.132849 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs" pod "eb1a97a5-e924-4286-a6c6-443f57262f7c" (UID: "eb1a97a5-e924-4286-a6c6-443f57262f7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:34:52.151135 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.151058 4885 scope.go:122] "RemoveContainer" containerID="bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033" Nov 03 18:34:52.151609 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:34:52.151564 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033\": container with ID starting with bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033 not found: ID does not exist" containerID="bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033" Nov 03 18:34:52.151737 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.151604 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033"} err="failed to get container status \"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033\": rpc error: code = NotFound desc = could not find container \"bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033\": container with ID starting with bdde5b7e3508527339f98f4603d2e219c840b24c0610438cf4cd2d1f7fc3f033 not found: ID does not exist" Nov 03 18:34:52.229927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.229893 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:34:52.229927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.229924 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a97a5-e924-4286-a6c6-443f57262f7c-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:34:52.229927 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.229933 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb1a97a5-e924-4286-a6c6-443f57262f7c-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:34:52.230183 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.229942 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgh8\" (UniqueName: \"kubernetes.io/projected/eb1a97a5-e924-4286-a6c6-443f57262f7c-kube-api-access-cbgh8\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:34:52.557291 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.557254 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:34:52.610382 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:52.610340 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-57d07b-7d9487bd68-pn2bf"] Nov 03 18:34:53.661708 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:53.661669 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1a97a5-e924-4286-a6c6-443f57262f7c" path="/var/lib/kubelet/pods/eb1a97a5-e924-4286-a6c6-443f57262f7c/volumes" Nov 03 18:34:55.547765 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:34:55.547726 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:35:16.052179 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:16.052137 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh"] Nov 03 18:35:16.103391 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:16.103357 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-mk8fh"] Nov 03 18:35:17.661526 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:17.661488 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1" path="/var/lib/kubelet/pods/0eb96bb5-1005-4c58-8cb5-b0ec3f2790b1/volumes" Nov 03 18:35:18.605548 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:35:18.605510 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:35:46.552805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:46.552773 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:35:48.641094 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:35:48.633795 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:35:50.960162 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:50.960124 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:35:50.960684 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:35:50.960403 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:36:13.972551 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:36:13.972462 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:36:18.650094 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:36:18.648320 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:36:38.914704 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:36:38.914661 4885 scope.go:122] "RemoveContainer" containerID="3b44d20a91e6558d8bbda73efbaebf61b52ef5e024f623bdedca7c4346b47ef9" Nov 03 18:36:45.368616 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:36:45.368567 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:36:48.682960 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:36:48.682917 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:37:00.734780 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:00.734742 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:37:11.762276 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:11.762238 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:37:18.606587 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:37:18.606548 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:37:24.655115 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:24.655080 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:37:32.263589 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:32.263503 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:37:35.984296 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:35.984253 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:37:35.984954 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:35.984914 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:37:48.604668 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:37:48.604631 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:37:51.434865 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:37:51.434835 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:38:17.569084 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:17.569027 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:38:18.606098 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:38:18.606043 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:38:44.130159 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.130126 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:44.130558 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.130461 4885 cpu_manager.go:414] "containerMap: removing container" podUID="eb1a97a5-e924-4286-a6c6-443f57262f7c" containerName="maas-api" Nov 03 18:38:44.130558 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.130471 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1a97a5-e924-4286-a6c6-443f57262f7c" containerName="maas-api" Nov 03 18:38:44.130558 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.130531 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1a97a5-e924-4286-a6c6-443f57262f7c" containerName="maas-api" Nov 03 18:38:44.133349 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.133331 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.188348 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.188306 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:44.226288 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.226253 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz75r\" (UniqueName: \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.226478 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.226305 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.238341 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.238317 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48c17a\"" Nov 03 18:38:44.238467 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.238379 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48c17a-dockercfg-px62t\"" Nov 03 18:38:44.327227 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.327196 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz75r\" (UniqueName: \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.327435 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.327243 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.327892 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.327873 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.390551 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.390474 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz75r\" (UniqueName: \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\") pod \"payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.445735 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.445694 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:44.634119 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.634087 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:44.814677 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:38:44.814648 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cca6bb_36fe_447d_b075_f17a8aac0b4e.slice/crio-7fc98854ddd7357997ec71e62b7f3e2435a40c86415fcf4a4cf58ce4f4cc3641 WatchSource:0}: Error finding container 7fc98854ddd7357997ec71e62b7f3e2435a40c86415fcf4a4cf58ce4f4cc3641: Status 404 returned error can't find the container with id 7fc98854ddd7357997ec71e62b7f3e2435a40c86415fcf4a4cf58ce4f4cc3641 Nov 03 18:38:44.815123 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:44.815098 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:45.251133 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:45.251103 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" event={"ID":"65cca6bb-36fe-447d-b075-f17a8aac0b4e","Type":"ContainerStarted","Data":"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f"} Nov 03 18:38:45.251621 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:45.251159 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" event={"ID":"65cca6bb-36fe-447d-b075-f17a8aac0b4e","Type":"ContainerStarted","Data":"7fc98854ddd7357997ec71e62b7f3e2435a40c86415fcf4a4cf58ce4f4cc3641"} Nov 03 18:38:45.251621 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:45.251252 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:45.372527 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:45.372473 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" podStartSLOduration=1.37245852 podStartE2EDuration="1.37245852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:38:45.371227782 +0000 UTC m=+4867.825916808" watchObservedRunningTime="2026-11-03 18:38:45.37245852 +0000 UTC m=+4867.827147523" Nov 03 18:38:48.605661 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:38:48.605620 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:38:50.252100 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.252019 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" podUID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" containerName="payload-processing" containerID="cri-o://23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f" gracePeriod=25 Nov 03 18:38:50.490482 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.490450 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:50.581290 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.581192 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\") pod \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " Nov 03 18:38:50.581290 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.581264 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\" (UniqueName: \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\") pod \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\" (UID: \"65cca6bb-36fe-447d-b075-f17a8aac0b4e\") " Nov 03 18:38:50.581579 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.581555 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume" pod "65cca6bb-36fe-447d-b075-f17a8aac0b4e" (UID: "65cca6bb-36fe-447d-b075-f17a8aac0b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:50.583559 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.583536 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r" pod "65cca6bb-36fe-447d-b075-f17a8aac0b4e" (UID: "65cca6bb-36fe-447d-b075-f17a8aac0b4e"). InnerVolumeSpecName "kube-api-access-lz75r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:50.682365 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.682325 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz75r\" (UniqueName: \"kubernetes.io/projected/65cca6bb-36fe-447d-b075-f17a8aac0b4e-kube-api-access-lz75r\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:50.682365 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:50.682356 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cca6bb-36fe-447d-b075-f17a8aac0b4e-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:38:51.277331 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.277298 4885 generic.go:356] "Generic (PLEG): container finished" podID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" containerID="23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f" exitCode=0 Nov 03 18:38:51.277331 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.277338 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" Nov 03 18:38:51.277916 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.277388 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" event={"ID":"65cca6bb-36fe-447d-b075-f17a8aac0b4e","Type":"ContainerDied","Data":"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f"} Nov 03 18:38:51.277916 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.277450 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74" event={"ID":"65cca6bb-36fe-447d-b075-f17a8aac0b4e","Type":"ContainerDied","Data":"7fc98854ddd7357997ec71e62b7f3e2435a40c86415fcf4a4cf58ce4f4cc3641"} Nov 03 18:38:51.277916 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.277484 4885 scope.go:122] "RemoveContainer" containerID="23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f" Nov 03 18:38:51.286278 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.286253 4885 scope.go:122] "RemoveContainer" containerID="23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f" Nov 03 18:38:51.286577 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:38:51.286557 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f\": container with ID starting with 23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f not found: ID does not exist" containerID="23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f" Nov 03 18:38:51.286631 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.286586 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f"} err="failed to get container status \"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f\": rpc error: code = NotFound desc = could not find container \"23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f\": container with ID starting with 23e744e6ac450f42dc28ce3d73347e2786c05830e522a8b8ff289655d713743f not found: ID does not exist" Nov 03 18:38:51.445542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.445508 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:51.496904 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.496871 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-48c17a-574b588f7f-6nh74"] Nov 03 18:38:51.661442 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:38:51.661403 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" path="/var/lib/kubelet/pods/65cca6bb-36fe-447d-b075-f17a8aac0b4e/volumes" Nov 03 18:39:00.107641 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.107549 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:39:00.108131 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.107869 4885 cpu_manager.go:414] "containerMap: removing container" podUID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" containerName="payload-processing" Nov 03 18:39:00.108131 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.107880 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" containerName="payload-processing" Nov 03 18:39:00.108131 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.107964 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cca6bb-36fe-447d-b075-f17a8aac0b4e" containerName="payload-processing" Nov 03 18:39:00.112275 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.112254 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.166127 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.166093 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:39:00.215955 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.215912 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-48c17a\"" Nov 03 18:39:00.215955 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.215927 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-48c17a-dockercfg-wnd7t\"" Nov 03 18:39:00.269775 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.269735 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.269954 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.269916 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrbj\" (UniqueName: \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.371426 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.371395 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrbj\" (UniqueName: \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.371602 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.371455 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.372152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.372135 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.436154 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.436113 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrbj\" (UniqueName: \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\") pod \"payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.728817 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.728724 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:00.951302 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:00.951269 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:39:01.105397 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:39:01.105363 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cda4ff4_be03_4843_b3c3_403b0ea15680.slice/crio-04c2d5d4983195489ffb965baedb03d9ae411117c958c332f7b3608d6e1aba19 WatchSource:0}: Error finding container 04c2d5d4983195489ffb965baedb03d9ae411117c958c332f7b3608d6e1aba19: Status 404 returned error can't find the container with id 04c2d5d4983195489ffb965baedb03d9ae411117c958c332f7b3608d6e1aba19 Nov 03 18:39:01.108420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:01.108347 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:39:01.115562 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:01.115534 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:39:01.323442 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:01.323405 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" event={"ID":"1cda4ff4-be03-4843-b3c3-403b0ea15680","Type":"ContainerStarted","Data":"04c2d5d4983195489ffb965baedb03d9ae411117c958c332f7b3608d6e1aba19"} Nov 03 18:39:04.337749 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:04.337710 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" event={"ID":"1cda4ff4-be03-4843-b3c3-403b0ea15680","Type":"ContainerStarted","Data":"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40"} Nov 03 18:39:04.338268 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:04.337795 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:04.458292 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:04.458231 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" podStartSLOduration=2.165634199 podStartE2EDuration="4.458218247s" totalImagesPullingTime="2.292584048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:04.456776736 +0000 UTC m=+4886.911465761" watchObservedRunningTime="2026-11-03 18:39:04.458218247 +0000 UTC m=+4886.912907272" Nov 03 18:39:15.343626 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:15.343591 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:39:18.605165 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:39:18.605120 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:39:32.039505 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.039472 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:39:32.042661 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.042645 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.099281 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.099252 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4ee65c\"/\"openshift-service-ca.crt\"" Nov 03 18:39:32.150328 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.150289 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4ee65c\"/\"kube-root-ca.crt\"" Nov 03 18:39:32.150533 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.150354 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4ee65c\"/\"default-dockercfg-8mdrq\"" Nov 03 18:39:32.150533 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.150424 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4ee65c\"/\"ipp-route-4ee65c-kserve-self-signed-certs\"" Nov 03 18:39:32.151743 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.151718 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:39:32.160935 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.160912 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ns2\" (UniqueName: \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.161043 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.160942 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.161043 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.160964 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.161043 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.160982 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.161164 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.161101 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.161164 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.161136 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262380 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ns2\" (UniqueName: \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262420 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262415 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262683 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262439 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262683 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262461 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262808 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262748 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262808 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262796 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.262918 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.262811 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.263127 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.263104 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.263250 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.263124 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.264842 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.264819 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.265167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.265151 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.325091 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.325009 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ns2\" (UniqueName: \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\") pod \"ipp-route-4ee65c-kserve-65c994ffbf-tnvw7\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.366947 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.366918 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:32.740930 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:32.740903 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:39:33.469605 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:33.469569 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" event={"ID":"9ff966d8-32a9-4bf6-b9a8-047869a264ec","Type":"ContainerStarted","Data":"99a30848e5b409fa2706e586536a7096fd540ea0b4eba462772e807d81e49f3a"} Nov 03 18:39:33.469605 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:33.469612 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" event={"ID":"9ff966d8-32a9-4bf6-b9a8-047869a264ec","Type":"ContainerStarted","Data":"f09a194ec3938381a014cca2cafd317b822785518a9bf362222d5156b9d676f8"} Nov 03 18:39:33.589451 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:33.589393 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" podStartSLOduration=2.589378159 podStartE2EDuration="2.589378159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:33.588498673 +0000 UTC m=+4916.043187711" watchObservedRunningTime="2026-11-03 18:39:33.589378159 +0000 UTC m=+4916.044067181" Nov 03 18:39:34.367708 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:34.367673 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:36.738117 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:36.738082 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:39:42.368109 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:42.368059 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:42.380480 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:42.380455 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:42.518136 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:39:42.518107 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:39:48.605214 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:39:48.605175 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:40:18.605590 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:40:18.605543 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:40:22.077234 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:22.077197 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:40:36.612703 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:36.612618 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:40:48.605127 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:40:48.605087 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:40:51.633934 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.633896 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" podUID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" containerName="main" containerID="cri-o://99a30848e5b409fa2706e586536a7096fd540ea0b4eba462772e807d81e49f3a" gracePeriod=45 Nov 03 18:40:51.825564 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.825528 4885 generic.go:356] "Generic (PLEG): container finished" podID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" containerID="99a30848e5b409fa2706e586536a7096fd540ea0b4eba462772e807d81e49f3a" exitCode=0 Nov 03 18:40:51.825740 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.825591 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" event={"ID":"9ff966d8-32a9-4bf6-b9a8-047869a264ec","Type":"ContainerDied","Data":"99a30848e5b409fa2706e586536a7096fd540ea0b4eba462772e807d81e49f3a"} Nov 03 18:40:51.878215 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.878193 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:40:51.971350 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971271 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971350 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971326 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971374 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971390 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971418 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\" (UniqueName: \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971542 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971458 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\") pod \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\" (UID: \"9ff966d8-32a9-4bf6-b9a8-047869a264ec\") " Nov 03 18:40:51.971710 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971574 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:51.971710 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971692 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:51.971790 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971708 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:51.971790 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.971723 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:51.973787 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.973763 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "kube-api-access-l5ns2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:51.973787 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.973777 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:40:51.973787 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:51.973766 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs" pod "9ff966d8-32a9-4bf6-b9a8-047869a264ec" (UID: "9ff966d8-32a9-4bf6-b9a8-047869a264ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:52.073165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.073132 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:52.073165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.073159 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:52.073165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.073168 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ns2\" (UniqueName: \"kubernetes.io/projected/9ff966d8-32a9-4bf6-b9a8-047869a264ec-kube-api-access-l5ns2\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:52.073397 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.073178 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:52.073397 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.073186 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff966d8-32a9-4bf6-b9a8-047869a264ec-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:40:52.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.831655 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" Nov 03 18:40:52.831679 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.831666 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7" event={"ID":"9ff966d8-32a9-4bf6-b9a8-047869a264ec","Type":"ContainerDied","Data":"f09a194ec3938381a014cca2cafd317b822785518a9bf362222d5156b9d676f8"} Nov 03 18:40:52.832204 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.831711 4885 scope.go:122] "RemoveContainer" containerID="99a30848e5b409fa2706e586536a7096fd540ea0b4eba462772e807d81e49f3a" Nov 03 18:40:52.999234 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:52.999198 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:40:53.050516 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:53.050486 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4ee65c/ipp-route-4ee65c-kserve-65c994ffbf-tnvw7"] Nov 03 18:40:53.665839 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:40:53.665803 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" path="/var/lib/kubelet/pods/9ff966d8-32a9-4bf6-b9a8-047869a264ec/volumes" Nov 03 18:41:13.748793 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.748760 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2"] Nov 03 18:41:13.749190 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.749089 4885 cpu_manager.go:414] "containerMap: removing container" podUID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" containerName="main" Nov 03 18:41:13.749190 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.749099 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" containerName="main" Nov 03 18:41:13.749190 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.749174 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ff966d8-32a9-4bf6-b9a8-047869a264ec" containerName="main" Nov 03 18:41:13.752042 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.752027 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.807041 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.806964 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2"] Nov 03 18:41:13.856659 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.856621 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:41:13.856659 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.856642 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:41:13.856659 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.856622 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:41:13.857533 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.857514 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\"" Nov 03 18:41:13.872921 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.872894 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.873027 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.872928 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.873027 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.872959 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bg7\" (UniqueName: \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.974543 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.974508 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.974723 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.974560 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bg7\" (UniqueName: \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.974723 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.974623 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.975342 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.975272 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:13.977021 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:13.976998 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:14.037426 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:14.037366 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bg7\" (UniqueName: \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\") pod \"maas-api-revoke-sub-bc7e7c87-cm7w2\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:14.064963 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:14.064927 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:14.432696 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:41:14.432664 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355b36cd_5a2b_4efc_8ce5_f464c1777a53.slice/crio-fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f WatchSource:0}: Error finding container fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f: Status 404 returned error can't find the container with id fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f Nov 03 18:41:14.433615 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:14.433587 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2"] Nov 03 18:41:14.941446 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:14.941401 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" event={"ID":"355b36cd-5a2b-4efc-8ce5-f464c1777a53","Type":"ContainerStarted","Data":"95ad87ff3e22dda794e5a63a0aec344066f94ee025c91349a0ae4eff5d1bab00"} Nov 03 18:41:14.941917 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:14.941474 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" event={"ID":"355b36cd-5a2b-4efc-8ce5-f464c1777a53","Type":"ContainerStarted","Data":"fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f"} Nov 03 18:41:15.061750 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:15.061645 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" podStartSLOduration=2.061631809 podStartE2EDuration="2.061631809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:41:15.058977065 +0000 UTC m=+5017.513666090" watchObservedRunningTime="2026-11-03 18:41:15.061631809 +0000 UTC m=+5017.516320834" Nov 03 18:41:15.947445 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:15.947409 4885 generic.go:356] "Generic (PLEG): container finished" podID="355b36cd-5a2b-4efc-8ce5-f464c1777a53" containerID="95ad87ff3e22dda794e5a63a0aec344066f94ee025c91349a0ae4eff5d1bab00" exitCode=0 Nov 03 18:41:15.947916 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:15.947496 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" event={"ID":"355b36cd-5a2b-4efc-8ce5-f464c1777a53","Type":"ContainerDied","Data":"95ad87ff3e22dda794e5a63a0aec344066f94ee025c91349a0ae4eff5d1bab00"} Nov 03 18:41:17.089469 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.089444 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:17.202847 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.202798 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\") pod \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " Nov 03 18:41:17.203101 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.202882 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\") pod \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " Nov 03 18:41:17.203101 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.202936 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\" (UniqueName: \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\") pod \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\" (UID: \"355b36cd-5a2b-4efc-8ce5-f464c1777a53\") " Nov 03 18:41:17.203325 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.203297 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca" pod "355b36cd-5a2b-4efc-8ce5-f464c1777a53" (UID: "355b36cd-5a2b-4efc-8ce5-f464c1777a53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:41:17.205221 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.205195 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7" pod "355b36cd-5a2b-4efc-8ce5-f464c1777a53" (UID: "355b36cd-5a2b-4efc-8ce5-f464c1777a53"). InnerVolumeSpecName "kube-api-access-87bg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:41:17.205221 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.205207 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca" pod "355b36cd-5a2b-4efc-8ce5-f464c1777a53" (UID: "355b36cd-5a2b-4efc-8ce5-f464c1777a53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:41:17.304493 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.304457 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:41:17.304493 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.304495 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/355b36cd-5a2b-4efc-8ce5-f464c1777a53-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:41:17.304723 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.304510 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bg7\" (UniqueName: \"kubernetes.io/projected/355b36cd-5a2b-4efc-8ce5-f464c1777a53-kube-api-access-87bg7\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:41:17.581274 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.581184 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:41:17.958096 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.958037 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" Nov 03 18:41:17.958294 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.958105 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2" event={"ID":"355b36cd-5a2b-4efc-8ce5-f464c1777a53","Type":"ContainerDied","Data":"fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f"} Nov 03 18:41:17.958294 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:17.958146 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac54d6c34b06eea6e55f299b3a72a06e2f8f5632309e023c32e04a06a0d878f" Nov 03 18:41:18.605429 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:41:18.605389 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:41:32.872339 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:32.872303 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:41:48.604558 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:41:48.604523 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:41:54.562740 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.562706 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:41:54.563258 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.562924 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" podUID="1cda4ff4-be03-4843-b3c3-403b0ea15680" containerName="payload-pre-processing" containerID="cri-o://b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40" gracePeriod=30 Nov 03 18:41:54.811056 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.811028 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:41:54.839267 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.839198 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\" (UniqueName: \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\") pod \"1cda4ff4-be03-4843-b3c3-403b0ea15680\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " Nov 03 18:41:54.839267 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.839237 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\") pod \"1cda4ff4-be03-4843-b3c3-403b0ea15680\" (UID: \"1cda4ff4-be03-4843-b3c3-403b0ea15680\") " Nov 03 18:41:54.839613 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.839585 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume" pod "1cda4ff4-be03-4843-b3c3-403b0ea15680" (UID: "1cda4ff4-be03-4843-b3c3-403b0ea15680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:41:54.841453 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.841436 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj" pod "1cda4ff4-be03-4843-b3c3-403b0ea15680" (UID: "1cda4ff4-be03-4843-b3c3-403b0ea15680"). InnerVolumeSpecName "kube-api-access-wkrbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:41:54.939891 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.939857 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkrbj\" (UniqueName: \"kubernetes.io/projected/1cda4ff4-be03-4843-b3c3-403b0ea15680-kube-api-access-wkrbj\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:41:54.939891 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:54.939886 4885 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cda4ff4-be03-4843-b3c3-403b0ea15680-plugins-config-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:41:55.127543 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.127508 4885 generic.go:356] "Generic (PLEG): container finished" podID="1cda4ff4-be03-4843-b3c3-403b0ea15680" containerID="b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40" exitCode=0 Nov 03 18:41:55.127724 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.127564 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" Nov 03 18:41:55.127724 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.127569 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" event={"ID":"1cda4ff4-be03-4843-b3c3-403b0ea15680","Type":"ContainerDied","Data":"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40"} Nov 03 18:41:55.127724 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.127641 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc" event={"ID":"1cda4ff4-be03-4843-b3c3-403b0ea15680","Type":"ContainerDied","Data":"04c2d5d4983195489ffb965baedb03d9ae411117c958c332f7b3608d6e1aba19"} Nov 03 18:41:55.127724 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.127664 4885 scope.go:122] "RemoveContainer" containerID="b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40" Nov 03 18:41:55.135923 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.135901 4885 scope.go:122] "RemoveContainer" containerID="b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40" Nov 03 18:41:55.136239 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:41:55.136220 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40\": container with ID starting with b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40 not found: ID does not exist" containerID="b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40" Nov 03 18:41:55.136288 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.136250 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40"} err="failed to get container status \"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40\": rpc error: code = NotFound desc = could not find container \"b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40\": container with ID starting with b1b907ecbc6fd1f605e1750093ae051ec027fa1378fc227a290a7f73f8af9c40 not found: ID does not exist" Nov 03 18:41:55.295125 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.295091 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:41:55.298634 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.298609 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-48c17a-dcc965dc9-l5dlc"] Nov 03 18:41:55.660516 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:55.660486 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cda4ff4-be03-4843-b3c3-403b0ea15680" path="/var/lib/kubelet/pods/1cda4ff4-be03-4843-b3c3-403b0ea15680/volumes" Nov 03 18:41:58.362346 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:41:58.362310 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:42:18.605942 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:42:18.605861 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:42:48.604948 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:42:48.604906 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:42:54.368026 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:42:54.367993 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:43:18.606255 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:43:18.606214 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:43:48.604542 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:43:48.604506 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:44:18.605665 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:44:18.605626 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:44:27.333642 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:44:27.333603 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:44:38.719760 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:44:38.719727 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:44:47.034533 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:44:47.034493 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:44:48.604535 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:44:48.604491 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:45:02.633596 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:02.633517 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:45:13.039738 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:13.039701 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:45:14.497444 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497408 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt"] Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497741 4885 cpu_manager.go:414] "containerMap: removing container" podUID="1cda4ff4-be03-4843-b3c3-403b0ea15680" containerName="payload-pre-processing" Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497751 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cda4ff4-be03-4843-b3c3-403b0ea15680" containerName="payload-pre-processing" Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497766 4885 cpu_manager.go:414] "containerMap: removing container" podUID="355b36cd-5a2b-4efc-8ce5-f464c1777a53" containerName="revoke-keys" Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497771 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355b36cd-5a2b-4efc-8ce5-f464c1777a53" containerName="revoke-keys" Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497822 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cda4ff4-be03-4843-b3c3-403b0ea15680" containerName="payload-pre-processing" Nov 03 18:45:14.497851 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.497831 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="355b36cd-5a2b-4efc-8ce5-f464c1777a53" containerName="revoke-keys" Nov 03 18:45:14.500411 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.500394 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.556878 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.556847 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt"] Nov 03 18:45:14.593148 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.593111 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.593316 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.593161 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcfv\" (UniqueName: \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.593316 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.593191 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.606086 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.606028 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:45:14.606086 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.606048 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:45:14.606086 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.606036 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:45:14.606086 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.606056 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-42b9q\"" Nov 03 18:45:14.694323 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.694283 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcfv\" (UniqueName: \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.694530 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.694334 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.694530 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.694397 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.695156 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.695136 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.696994 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.696972 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.757042 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.756965 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcfv\" (UniqueName: \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\") pod \"maas-api-revoke-sub-2e1f5b5f-56rgt\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:14.813601 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:14.813565 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:15.179504 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:45:15.179470 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5959a58_14f8_4e8b_bf59_a8dff31fe0c0.slice/crio-a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006 WatchSource:0}: Error finding container a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006: Status 404 returned error can't find the container with id a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006 Nov 03 18:45:15.181271 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:15.181239 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt"] Nov 03 18:45:16.110374 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:16.110344 4885 generic.go:356] "Generic (PLEG): container finished" podID="b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" containerID="7e82ad5edb3637f7ebc3c3784c7b41c857ac8578c4a283fe537026da7b95b79a" exitCode=0 Nov 03 18:45:16.110781 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:16.110431 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" event={"ID":"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0","Type":"ContainerDied","Data":"7e82ad5edb3637f7ebc3c3784c7b41c857ac8578c4a283fe537026da7b95b79a"} Nov 03 18:45:16.110781 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:16.110496 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" event={"ID":"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0","Type":"ContainerStarted","Data":"a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006"} Nov 03 18:45:17.242303 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.242275 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:17.420667 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.420634 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\") pod \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " Nov 03 18:45:17.420860 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.420706 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\") pod \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " Nov 03 18:45:17.420860 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.420760 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\" (UniqueName: \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\") pod \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\" (UID: \"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0\") " Nov 03 18:45:17.421107 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.421055 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca" pod "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" (UID: "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:45:17.423080 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.423035 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca" pod "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" (UID: "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:45:17.423200 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.423079 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv" pod "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" (UID: "b5959a58-14f8-4e8b-bf59-a8dff31fe0c0"). InnerVolumeSpecName "kube-api-access-mrcfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:45:17.521919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.521887 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-openshift-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:17.521919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.521914 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrcfv\" (UniqueName: \"kubernetes.io/projected/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-kube-api-access-mrcfv\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:17.521919 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:17.521924 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0-maas-api-xks-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:45:18.119788 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:18.119754 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" Nov 03 18:45:18.120006 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:18.119786 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt" event={"ID":"b5959a58-14f8-4e8b-bf59-a8dff31fe0c0","Type":"ContainerDied","Data":"a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006"} Nov 03 18:45:18.120006 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:18.119847 4885 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a37741f3fb2b197a53ce9989034c4b1285abbf952b3e63ec7f812ab79f658006" Nov 03 18:45:18.605892 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:45:18.605803 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:45:27.246165 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:27.246127 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:45:44.235001 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:45:44.234969 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:45:48.608626 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:45:48.608508 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:46:16.338914 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:16.338879 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:46:18.053582 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:18.053543 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2"] Nov 03 18:46:18.107548 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:18.107509 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc7e7c87-cm7w2"] Nov 03 18:46:18.648804 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:46:18.648764 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:46:19.668939 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:19.668900 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355b36cd-5a2b-4efc-8ce5-f464c1777a53" path="/var/lib/kubelet/pods/355b36cd-5a2b-4efc-8ce5-f464c1777a53/volumes" Nov 03 18:46:26.953955 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:26.953918 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:46:35.985719 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:35.985604 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:46:35.986488 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:35.986452 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:46:38.973685 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:46:38.973651 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:46:48.605116 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:46:48.605054 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:47:02.644274 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:02.644240 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:12.154050 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:12.154014 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:18.605392 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:47:18.605352 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:47:20.147008 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:20.146977 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:28.737666 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:28.737628 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:31.688270 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:31.688221 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:47:31.688716 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:31.688281 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:47:31.766093 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:31.765183 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:47:31.766093 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:31.765271 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:47:38.028941 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:38.028900 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:39.012236 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:39.012209 4885 scope.go:122] "RemoveContainer" containerID="95ad87ff3e22dda794e5a63a0aec344066f94ee025c91349a0ae4eff5d1bab00" Nov 03 18:47:46.641953 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:46.641914 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:47:48.666281 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:47:48.666237 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:47:50.954389 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:50.954343 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:47:50.954810 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:50.954416 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:47:55.849525 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:47:55.849485 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:48:13.156812 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:13.156728 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:48:16.686234 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:16.686198 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-927sl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:48:16.686646 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:16.686252 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:48:16.758253 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:16.758212 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-csqjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:48:16.758563 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:16.758532 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:48:18.605940 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:48:18.605897 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:48:23.957321 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:23.957288 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:48:38.450939 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:38.450905 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:48:48.229798 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:48.229760 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:48:48.605307 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:48:48.605223 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:48:56.268319 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:48:56.268283 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:49:16.651879 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:49:16.651842 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:49:18.682517 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:49:18.682459 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:49:27.460448 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:49:27.460416 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:49:46.047632 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:49:46.047598 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:49:48.700092 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:49:48.698361 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:49:59.158099 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:49:59.158053 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:50:10.250224 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:10.250185 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:50:18.052023 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:18.051989 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt"] Nov 03 18:50:18.103531 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:18.103498 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-56rgt"] Nov 03 18:50:18.608630 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:50:18.608593 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:50:19.661088 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:19.661036 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" path="/var/lib/kubelet/pods/b5959a58-14f8-4e8b-bf59-a8dff31fe0c0/volumes" Nov 03 18:50:23.578102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.575319 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:50:23.578102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.576136 4885 cpu_manager.go:414] "containerMap: removing container" podUID="b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" containerName="revoke-keys" Nov 03 18:50:23.578102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.576154 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" containerName="revoke-keys" Nov 03 18:50:23.578102 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.576362 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5959a58-14f8-4e8b-bf59-a8dff31fe0c0" containerName="revoke-keys" Nov 03 18:50:23.582168 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.582142 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.657005 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.656967 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-27a21b\"/\"default-dockercfg-kzw4h\"" Nov 03 18:50:23.699995 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.699960 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.700177 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.700005 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.700177 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.700107 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.700177 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.700136 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wl4\" (UniqueName: \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.700287 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.700175 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.700287 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.700216 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.708562 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.708532 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-27a21b\"/\"kube-root-ca.crt\"" Nov 03 18:50:23.708562 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.708533 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-27a21b\"/\"openshift-service-ca.crt\"" Nov 03 18:50:23.708722 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.708581 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-27a21b\"/\"simulator-single-main-27a21b-kserve-self-signed-certs\"" Nov 03 18:50:23.710008 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.709990 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:50:23.801651 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801619 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.801854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801660 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.801854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801695 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.801854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801712 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wl4\" (UniqueName: \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.801854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801736 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.801854 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.801751 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.802167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.802109 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.802167 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.802153 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.802270 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.802253 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.804132 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.804108 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.804384 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.804367 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.865056 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.864972 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wl4\" (UniqueName: \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\") pod \"simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:23.899903 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:23.899863 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:24.271770 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:50:24.271739 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6e11f82_bf05_477e_890d_03ffd565f5dc.slice/crio-c9be6af0aa0d3003259c0c761a5f5b011a2ff0d02c4fe9cb36a1e57dd7a4257f WatchSource:0}: Error finding container c9be6af0aa0d3003259c0c761a5f5b011a2ff0d02c4fe9cb36a1e57dd7a4257f: Status 404 returned error can't find the container with id c9be6af0aa0d3003259c0c761a5f5b011a2ff0d02c4fe9cb36a1e57dd7a4257f Nov 03 18:50:24.272999 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:24.272973 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:50:24.699569 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:24.699531 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" event={"ID":"e6e11f82-bf05-477e-890d-03ffd565f5dc","Type":"ContainerStarted","Data":"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a"} Nov 03 18:50:24.699975 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:24.699580 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" event={"ID":"e6e11f82-bf05-477e-890d-03ffd565f5dc","Type":"ContainerStarted","Data":"c9be6af0aa0d3003259c0c761a5f5b011a2ff0d02c4fe9cb36a1e57dd7a4257f"} Nov 03 18:50:24.819835 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:24.819786 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" podStartSLOduration=1.819773114 podStartE2EDuration="1.819773114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:24.81878733 +0000 UTC m=+5567.273476355" watchObservedRunningTime="2026-11-03 18:50:24.819773114 +0000 UTC m=+5567.274462140" Nov 03 18:50:24.900113 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:24.900060 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:28.041653 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:28.041617 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:50:33.900749 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:33.900712 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:33.913377 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:33.913353 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:34.771393 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:34.771366 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:50:48.604936 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:50:48.604901 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:50:52.167630 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:50:52.167594 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:51:18.605160 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:51:18.605055 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:51:21.484882 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:21.484849 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:51:23.410655 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:23.410620 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:51:35.137802 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:35.137769 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:51:38.430747 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.430698 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" podUID="e6e11f82-bf05-477e-890d-03ffd565f5dc" containerName="main" containerID="cri-o://3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a" gracePeriod=45 Nov 03 18:51:38.684054 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.683991 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:51:38.771805 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771776 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772017 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771826 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772017 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771854 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772017 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771879 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772017 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771920 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\" (UniqueName: \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772017 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.771950 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\") pod \"e6e11f82-bf05-477e-890d-03ffd565f5dc\" (UID: \"e6e11f82-bf05-477e-890d-03ffd565f5dc\") " Nov 03 18:51:38.772358 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.772099 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:38.772358 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.772192 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:38.772358 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.772303 4885 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-home\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:38.772358 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.772303 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:38.772358 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.772320 4885 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-model-cache\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:38.774350 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.774323 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "kube-api-access-g9wl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:51:38.774350 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.774335 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:51:38.774487 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.774388 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm" pod "e6e11f82-bf05-477e-890d-03ffd565f5dc" (UID: "e6e11f82-bf05-477e-890d-03ffd565f5dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:51:38.873166 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.873138 4885 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-dshm\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:38.873166 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.873164 4885 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e11f82-bf05-477e-890d-03ffd565f5dc-tmp-dir\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:38.873166 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.873174 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wl4\" (UniqueName: \"kubernetes.io/projected/e6e11f82-bf05-477e-890d-03ffd565f5dc-kube-api-access-g9wl4\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:38.873388 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:38.873183 4885 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e11f82-bf05-477e-890d-03ffd565f5dc-tls-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:51:39.038699 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.038608 4885 generic.go:356] "Generic (PLEG): container finished" podID="e6e11f82-bf05-477e-890d-03ffd565f5dc" containerID="3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a" exitCode=0 Nov 03 18:51:39.038699 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.038651 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" Nov 03 18:51:39.038699 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.038663 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" event={"ID":"e6e11f82-bf05-477e-890d-03ffd565f5dc","Type":"ContainerDied","Data":"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a"} Nov 03 18:51:39.038939 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.038706 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf" event={"ID":"e6e11f82-bf05-477e-890d-03ffd565f5dc","Type":"ContainerDied","Data":"c9be6af0aa0d3003259c0c761a5f5b011a2ff0d02c4fe9cb36a1e57dd7a4257f"} Nov 03 18:51:39.038939 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.038724 4885 scope.go:122] "RemoveContainer" containerID="3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a" Nov 03 18:51:39.047732 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.047701 4885 scope.go:122] "RemoveContainer" containerID="3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a" Nov 03 18:51:39.048011 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:51:39.047992 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a\": container with ID starting with 3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a not found: ID does not exist" containerID="3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a" Nov 03 18:51:39.048114 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.048021 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a"} err="failed to get container status \"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a\": rpc error: code = NotFound desc = could not find container \"3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a\": container with ID starting with 3656a9be8863d0b6c386538458f73d1b9d0e5a8a24b492e57763ba27a744554a not found: ID does not exist" Nov 03 18:51:39.052723 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.052705 4885 scope.go:122] "RemoveContainer" containerID="7e82ad5edb3637f7ebc3c3784c7b41c857ac8578c4a283fe537026da7b95b79a" Nov 03 18:51:39.206952 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.206910 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:51:39.210411 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.210383 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-27a21b/simulator-single-main-27a21b-kserve-5cc8b5999c-nwfwf"] Nov 03 18:51:39.662384 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:51:39.662350 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6e11f82-bf05-477e-890d-03ffd565f5dc" path="/var/lib/kubelet/pods/e6e11f82-bf05-477e-890d-03ffd565f5dc/volumes" Nov 03 18:51:48.605468 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:51:48.605426 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:52:18.604528 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:52:18.604489 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:52:18.754204 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:18.754169 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:52:20.978826 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:20.978776 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:52:20.979431 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:20.979387 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:52:23.108829 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:23.108796 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 18:52:23.109282 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:23.108986 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" podUID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" containerName="manager" containerID="cri-o://219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d" gracePeriod=10 Nov 03 18:52:23.960510 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:23.960480 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 18:52:24.096660 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.096585 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\") pod \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " Nov 03 18:52:24.096806 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.096688 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\" (UniqueName: \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\") pod \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\" (UID: \"a25335e8-e46d-4757-9cf4-0b36465d0b8e\") " Nov 03 18:52:24.097038 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.097002 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume" pod "a25335e8-e46d-4757-9cf4-0b36465d0b8e" (UID: "a25335e8-e46d-4757-9cf4-0b36465d0b8e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:52:24.098989 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.098966 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m" pod "a25335e8-e46d-4757-9cf4-0b36465d0b8e" (UID: "a25335e8-e46d-4757-9cf4-0b36465d0b8e"). InnerVolumeSpecName "kube-api-access-sxq6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:52:24.197883 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.197847 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxq6m\" (UniqueName: \"kubernetes.io/projected/a25335e8-e46d-4757-9cf4-0b36465d0b8e-kube-api-access-sxq6m\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:24.197883 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.197881 4885 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a25335e8-e46d-4757-9cf4-0b36465d0b8e-extensions-socket-volume\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:52:24.253835 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.253798 4885 generic.go:356] "Generic (PLEG): container finished" podID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" containerID="219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d" exitCode=0 Nov 03 18:52:24.253999 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.253852 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" Nov 03 18:52:24.253999 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.253879 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" event={"ID":"a25335e8-e46d-4757-9cf4-0b36465d0b8e","Type":"ContainerDied","Data":"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d"} Nov 03 18:52:24.253999 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.253949 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb" event={"ID":"a25335e8-e46d-4757-9cf4-0b36465d0b8e","Type":"ContainerDied","Data":"4d080040a01a3c82aad0e37139441c6db3047b03f7188f98821089e937ef99fe"} Nov 03 18:52:24.253999 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.253971 4885 scope.go:122] "RemoveContainer" containerID="219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d" Nov 03 18:52:24.266729 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.266709 4885 scope.go:122] "RemoveContainer" containerID="219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d" Nov 03 18:52:24.266976 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:52:24.266956 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d\": container with ID starting with 219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d not found: ID does not exist" containerID="219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d" Nov 03 18:52:24.267048 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.266990 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d"} err="failed to get container status \"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d\": rpc error: code = NotFound desc = could not find container \"219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d\": container with ID starting with 219810e0b43e8adcfba75e96b51eddbf03ab9aae78f1b6742105be7fcbf0858d not found: ID does not exist" Nov 03 18:52:24.422560 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.422524 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 18:52:24.426959 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:24.426925 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wdgfb"] Nov 03 18:52:25.660056 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:52:25.660019 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" path="/var/lib/kubelet/pods/a25335e8-e46d-4757-9cf4-0b36465d0b8e/volumes" Nov 03 18:52:48.604758 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:52:48.604713 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:53:18.604523 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:53:18.604483 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:53:26.746389 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746346 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv"] Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746660 4885 cpu_manager.go:414] "containerMap: removing container" podUID="e6e11f82-bf05-477e-890d-03ffd565f5dc" containerName="main" Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746671 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6e11f82-bf05-477e-890d-03ffd565f5dc" containerName="main" Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746682 4885 cpu_manager.go:414] "containerMap: removing container" podUID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" containerName="manager" Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746687 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" containerName="manager" Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746752 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6e11f82-bf05-477e-890d-03ffd565f5dc" containerName="main" Nov 03 18:53:26.746786 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.746759 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25335e8-e46d-4757-9cf4-0b36465d0b8e" containerName="manager" Nov 03 18:53:26.749663 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.749648 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:26.797636 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.797607 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv"] Nov 03 18:53:26.847269 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.847237 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f7pq2\"" Nov 03 18:53:26.934751 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.934725 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqhz\" (UniqueName: \"kubernetes.io/projected/47ac64e0-e994-4c7d-93ea-344eb970eb9d-kube-api-access-ndqhz\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:26.934922 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:26.934782 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ac64e0-e994-4c7d-93ea-344eb970eb9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.036258 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.036170 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqhz\" (UniqueName: \"kubernetes.io/projected/47ac64e0-e994-4c7d-93ea-344eb970eb9d-kube-api-access-ndqhz\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.036258 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.036253 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ac64e0-e994-4c7d-93ea-344eb970eb9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.036657 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.036634 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47ac64e0-e994-4c7d-93ea-344eb970eb9d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.098585 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.098550 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqhz\" (UniqueName: \"kubernetes.io/projected/47ac64e0-e994-4c7d-93ea-344eb970eb9d-kube-api-access-ndqhz\") pod \"kuadrant-operator-controller-manager-678d87766f-gqqlv\" (UID: \"47ac64e0-e994-4c7d-93ea-344eb970eb9d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.362518 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.362429 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:27.824918 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:27.824883 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv"] Nov 03 18:53:27.856237 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:53:27.856206 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ac64e0_e994_4c7d_93ea_344eb970eb9d.slice/crio-7edcdf8885b55c83ad64f0805b2c33a0bd20408a348d7c9637d01eb93bd04d62 WatchSource:0}: Error finding container 7edcdf8885b55c83ad64f0805b2c33a0bd20408a348d7c9637d01eb93bd04d62: Status 404 returned error can't find the container with id 7edcdf8885b55c83ad64f0805b2c33a0bd20408a348d7c9637d01eb93bd04d62 Nov 03 18:53:28.514265 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:28.514221 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" event={"ID":"47ac64e0-e994-4c7d-93ea-344eb970eb9d","Type":"ContainerStarted","Data":"d4a06dc58b955439f941f7d1e037c0be54c90ccc073d2d067456686b12abccbd"} Nov 03 18:53:28.514460 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:28.514285 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" event={"ID":"47ac64e0-e994-4c7d-93ea-344eb970eb9d","Type":"ContainerStarted","Data":"7edcdf8885b55c83ad64f0805b2c33a0bd20408a348d7c9637d01eb93bd04d62"} Nov 03 18:53:28.514460 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:28.514326 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:28.740840 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:28.740783 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" podStartSLOduration=2.740768924 podStartE2EDuration="2.740768924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:28.689147531 +0000 UTC m=+5751.143836555" watchObservedRunningTime="2026-11-03 18:53:28.740768924 +0000 UTC m=+5751.195457949" Nov 03 18:53:39.523486 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:39.523456 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-gqqlv" Nov 03 18:53:48.673109 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:53:48.670363 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:53:51.849550 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:53:51.849519 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:07.420311 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:07.420233 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:17.352495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:17.352461 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:18.605057 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:54:18.605018 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:54:25.852217 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:25.852186 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:27.240527 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.240495 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:27.243827 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.243810 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.274757 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.274729 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.274896 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.274787 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.274976 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.274947 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjkt\" (UniqueName: \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.275029 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.275013 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.298600 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.298575 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:54:27.350206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.350178 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 18:54:27.350206 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.350177 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:54:27.350417 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.350205 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r6cmh\"" Nov 03 18:54:27.350417 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.350227 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:54:27.352316 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.352293 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:27.375527 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.375500 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjkt\" (UniqueName: \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.375672 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.375546 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.375672 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.375611 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.375672 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.375665 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.376366 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.376345 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.378322 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.378299 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.378444 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.378304 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.438733 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.438703 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjkt\" (UniqueName: \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\") pod \"maas-api-d6b5cbff-vw4dj\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.556441 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.556358 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:27.926891 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:54:27.926863 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0b4e67_9c00_4dd7_b30a_a65c4101856b.slice/crio-7bc09424b582b980c493414684e2b7fb0a844eb09f4115d164f35985f99ff763 WatchSource:0}: Error finding container 7bc09424b582b980c493414684e2b7fb0a844eb09f4115d164f35985f99ff763: Status 404 returned error can't find the container with id 7bc09424b582b980c493414684e2b7fb0a844eb09f4115d164f35985f99ff763 Nov 03 18:54:27.928344 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.928324 4885 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:54:27.928620 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:27.928599 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:28.806176 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:28.806134 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" event={"ID":"7c0b4e67-9c00-4dd7-b30a-a65c4101856b","Type":"ContainerStarted","Data":"7bc09424b582b980c493414684e2b7fb0a844eb09f4115d164f35985f99ff763"} Nov 03 18:54:29.811179 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:29.811141 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" event={"ID":"7c0b4e67-9c00-4dd7-b30a-a65c4101856b","Type":"ContainerStarted","Data":"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518"} Nov 03 18:54:29.811582 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:29.811264 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:29.931990 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:29.931935 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" podStartSLOduration=2.052184928 podStartE2EDuration="2.931920456s" totalImagesPullingTime="879.735528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:29.931304266 +0000 UTC m=+5812.385993291" watchObservedRunningTime="2026-11-03 18:54:29.931920456 +0000 UTC m=+5812.386609481" Nov 03 18:54:35.821339 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:35.821309 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:35.927554 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:35.927527 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:35.927727 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:35.927691 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" podUID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" containerName="maas-api" containerID="cri-o://3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518" gracePeriod=30 Nov 03 18:54:35.980039 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:35.980010 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-669c98d779-lhckj"] Nov 03 18:54:35.983553 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:35.983524 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.044704 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.044674 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669c98d779-lhckj"] Nov 03 18:54:36.046629 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.046600 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls7vc\" (UniqueName: \"kubernetes.io/projected/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-kube-api-access-ls7vc\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.046725 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.046649 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-openshift-service-ca\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.046771 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.046747 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-metrics-certs\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.046857 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.046835 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-maas-api-tls\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.148244 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.148208 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls7vc\" (UniqueName: \"kubernetes.io/projected/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-kube-api-access-ls7vc\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.148413 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.148251 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-openshift-service-ca\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.148413 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.148282 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-metrics-certs\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.148413 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.148345 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-maas-api-tls\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.148912 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.148884 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-openshift-service-ca\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.151048 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.151021 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-maas-api-tls\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.151177 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.151021 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-metrics-certs\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.173465 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.173439 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:36.210807 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.210775 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls7vc\" (UniqueName: \"kubernetes.io/projected/3ad86a4a-6768-4e95-a0c0-eebfebb52d21-kube-api-access-ls7vc\") pod \"maas-api-669c98d779-lhckj\" (UID: \"3ad86a4a-6768-4e95-a0c0-eebfebb52d21\") " pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.249152 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249128 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\") pod \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " Nov 03 18:54:36.249311 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249162 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\" (UniqueName: \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\") pod \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " Nov 03 18:54:36.249311 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249202 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\") pod \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " Nov 03 18:54:36.249311 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249288 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\") pod \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\" (UID: \"7c0b4e67-9c00-4dd7-b30a-a65c4101856b\") " Nov 03 18:54:36.249495 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249464 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca" pod "7c0b4e67-9c00-4dd7-b30a-a65c4101856b" (UID: "7c0b4e67-9c00-4dd7-b30a-a65c4101856b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:54:36.249572 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.249554 4885 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-openshift-service-ca\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:36.251355 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.251334 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt" pod "7c0b4e67-9c00-4dd7-b30a-a65c4101856b" (UID: "7c0b4e67-9c00-4dd7-b30a-a65c4101856b"). InnerVolumeSpecName "kube-api-access-7cjkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:54:36.251432 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.251405 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls" pod "7c0b4e67-9c00-4dd7-b30a-a65c4101856b" (UID: "7c0b4e67-9c00-4dd7-b30a-a65c4101856b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:54:36.251549 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.251531 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs" pod "7c0b4e67-9c00-4dd7-b30a-a65c4101856b" (UID: "7c0b4e67-9c00-4dd7-b30a-a65c4101856b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:54:36.318602 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.318571 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:36.350332 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.350246 4885 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-maas-api-tls\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:36.350332 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.350286 4885 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-metrics-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:36.350332 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.350304 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjkt\" (UniqueName: \"kubernetes.io/projected/7c0b4e67-9c00-4dd7-b30a-a65c4101856b-kube-api-access-7cjkt\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:54:36.688825 ip-10-0-129-76 kubenswrapper[4885]: W1103 18:54:36.688790 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad86a4a_6768_4e95_a0c0_eebfebb52d21.slice/crio-b25d14911fa0ab05e979c0d358fb8109c50daea2925899d96c9f3c7a1a83cd20 WatchSource:0}: Error finding container b25d14911fa0ab05e979c0d358fb8109c50daea2925899d96c9f3c7a1a83cd20: Status 404 returned error can't find the container with id b25d14911fa0ab05e979c0d358fb8109c50daea2925899d96c9f3c7a1a83cd20 Nov 03 18:54:36.690832 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.690802 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669c98d779-lhckj"] Nov 03 18:54:36.841553 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.841509 4885 generic.go:356] "Generic (PLEG): container finished" podID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" containerID="3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518" exitCode=0 Nov 03 18:54:36.842015 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.841635 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" Nov 03 18:54:36.842015 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.841644 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" event={"ID":"7c0b4e67-9c00-4dd7-b30a-a65c4101856b","Type":"ContainerDied","Data":"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518"} Nov 03 18:54:36.842015 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.841714 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj" event={"ID":"7c0b4e67-9c00-4dd7-b30a-a65c4101856b","Type":"ContainerDied","Data":"7bc09424b582b980c493414684e2b7fb0a844eb09f4115d164f35985f99ff763"} Nov 03 18:54:36.842015 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.841744 4885 scope.go:122] "RemoveContainer" containerID="3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518" Nov 03 18:54:36.843346 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.843319 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" event={"ID":"3ad86a4a-6768-4e95-a0c0-eebfebb52d21","Type":"ContainerStarted","Data":"b25d14911fa0ab05e979c0d358fb8109c50daea2925899d96c9f3c7a1a83cd20"} Nov 03 18:54:36.851530 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.851511 4885 scope.go:122] "RemoveContainer" containerID="3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518" Nov 03 18:54:36.851829 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:54:36.851809 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518\": container with ID starting with 3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518 not found: ID does not exist" containerID="3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518" Nov 03 18:54:36.851906 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:36.851837 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518"} err="failed to get container status \"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518\": rpc error: code = NotFound desc = could not find container \"3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518\": container with ID starting with 3ac00b2806aeb30e977f2c309de5cc3ccb9a08f61e3f35250220712ed014a518 not found: ID does not exist" Nov 03 18:54:37.010531 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.010453 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:37.061502 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.061461 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d6b5cbff-vw4dj"] Nov 03 18:54:37.470471 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.470439 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:37.661403 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.661368 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" path="/var/lib/kubelet/pods/7c0b4e67-9c00-4dd7-b30a-a65c4101856b/volumes" Nov 03 18:54:37.851455 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.851361 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" event={"ID":"3ad86a4a-6768-4e95-a0c0-eebfebb52d21","Type":"ContainerStarted","Data":"e0ec98c32f680173ec53aa8b96138a2d93a8635f013813470a61a4ff50a23379"} Nov 03 18:54:37.851941 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.851475 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:37.973432 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:37.973366 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" podStartSLOduration=2.16915 podStartE2EDuration="2.973346341s" totalImagesPullingTime="804.196341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:37.972150062 +0000 UTC m=+5820.426839085" watchObservedRunningTime="2026-11-03 18:54:37.973346341 +0000 UTC m=+5820.428035371" Nov 03 18:54:43.862667 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:43.862639 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-669c98d779-lhckj" Nov 03 18:54:47.147778 ip-10-0-129-76 kubenswrapper[4885]: I1103 18:54:47.147742 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 18:54:48.605022 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:54:48.604984 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:55:18.604549 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:55:18.604505 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:55:48.605759 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:55:48.605718 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:56:18.605328 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:56:18.605283 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:56:48.605594 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:56:48.605555 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:57:18.605148 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:57:18.605043 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:57:48.605639 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:57:48.605598 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:58:18.604786 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:58:18.604747 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:58:48.604249 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:58:48.604214 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:59:18.604892 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:59:18.604852 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 18:59:48.604341 ip-10-0-129-76 kubenswrapper[4885]: E1103 18:59:48.604301 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:00:17.723181 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.723151 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 19:00:17.723640 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.723334 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" containerID="cri-o://f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734" gracePeriod=30 Nov 03 19:00:17.774832 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.774795 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 19:00:17.775089 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.775029 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" containerID="cri-o://e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec" gracePeriod=30 Nov 03 19:00:17.775944 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.775916 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 19:00:17.776168 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:17.776134 4885 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" containerID="cri-o://b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639" gracePeriod=30 Nov 03 19:00:18.605974 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:00:18.605931 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:00:19.937151 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:19.937116 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-zg2g2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 19:00:19.937553 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:19.937172 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 19:00:23.015715 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.015692 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 19:00:23.060568 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.060549 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 19:00:23.066781 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.066762 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 19:00:23.161222 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161189 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.161222 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161223 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161251 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161290 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161329 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161357 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161387 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161423 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161452 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161476 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161504 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161536 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161574 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161599 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161642 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161668 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161681 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161699 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\") pod \"90b428a3-01ca-46e2-a264-3415b03bf9f6\" (UID: \"90b428a3-01ca-46e2-a264-3415b03bf9f6\") " Nov 03 19:00:23.161761 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161702 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161771 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161811 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161840 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161881 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161920 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161963 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161994 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162027 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162057 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\") pod \"b9795628-59b5-44cb-baf3-d93ea91ce738\" (UID: \"b9795628-59b5-44cb-baf3-d93ea91ce738\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162118 4885 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\") pod \"69a32526-b77b-4ca2-872b-f5c928ddd077\" (UID: \"69a32526-b77b-4ca2-872b-f5c928ddd077\") " Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162420 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162441 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161776 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.161969 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162089 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162265 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162337 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162670 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162737 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 19:00:23.162923 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162870 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164029 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162995 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164029 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.162999 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164029 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.163043 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164029 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.163878 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164029 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.163883 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164324 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.164248 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.164566 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.164539 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "kube-api-access-mzp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.164775 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.164678 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:00:23.164863 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.164819 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.165951 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.165928 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.166746 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166704 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "kube-api-access-6bws2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.166746 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166704 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "kube-api-access-wtmk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.166903 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166768 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.166903 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166776 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 19:00:23.166903 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166773 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token" pod "90b428a3-01ca-46e2-a264-3415b03bf9f6" (UID: "90b428a3-01ca-46e2-a264-3415b03bf9f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 19:00:23.167026 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.166990 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo" pod "b9795628-59b5-44cb-baf3-d93ea91ce738" (UID: "b9795628-59b5-44cb-baf3-d93ea91ce738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:00:23.167099 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.167089 4885 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo" pod "69a32526-b77b-4ca2-872b-f5c928ddd077" (UID: "69a32526-b77b-4ca2-872b-f5c928ddd077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263418 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263447 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263461 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263473 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmk7\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-kube-api-access-wtmk7\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263483 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bws2\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-kube-api-access-6bws2\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263491 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263492 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263501 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263514 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263526 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263537 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263545 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263553 4885 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-data\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263561 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263569 4885 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-podinfo\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263576 4885 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-credential-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263588 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263601 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a32526-b77b-4ca2-872b-f5c928ddd077-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263612 4885 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzp5z\" (UniqueName: \"kubernetes.io/projected/b9795628-59b5-44cb-baf3-d93ea91ce738-kube-api-access-mzp5z\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263621 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b428a3-01ca-46e2-a264-3415b03bf9f6-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263629 4885 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a32526-b77b-4ca2-872b-f5c928ddd077-istio-token\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263637 4885 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9795628-59b5-44cb-baf3-d93ea91ce738-istio-envoy\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263645 4885 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-certs\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263652 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90b428a3-01ca-46e2-a264-3415b03bf9f6-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263661 4885 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9795628-59b5-44cb-baf3-d93ea91ce738-istiod-ca-cert\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.263814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.263669 4885 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90b428a3-01ca-46e2-a264-3415b03bf9f6-workload-socket\") on node \"ip-10-0-129-76.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 19:00:23.277193 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.277160 4885 generic.go:356] "Generic (PLEG): container finished" podID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerID="f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734" exitCode=0 Nov 03 19:00:23.277344 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.277211 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" Nov 03 19:00:23.277344 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.277249 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" event={"ID":"69a32526-b77b-4ca2-872b-f5c928ddd077","Type":"ContainerDied","Data":"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734"} Nov 03 19:00:23.277344 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.277300 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk" event={"ID":"69a32526-b77b-4ca2-872b-f5c928ddd077","Type":"ContainerDied","Data":"4b90ce8ccc1264ddca833d38606e162f8accf34072f0110356dc7da45236d9ae"} Nov 03 19:00:23.277344 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.277329 4885 scope.go:122] "RemoveContainer" containerID="f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734" Nov 03 19:00:23.278573 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.278535 4885 generic.go:356] "Generic (PLEG): container finished" podID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerID="b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639" exitCode=0 Nov 03 19:00:23.278649 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.278576 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" Nov 03 19:00:23.278649 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.278629 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" event={"ID":"90b428a3-01ca-46e2-a264-3415b03bf9f6","Type":"ContainerDied","Data":"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639"} Nov 03 19:00:23.278769 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.278673 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2" event={"ID":"90b428a3-01ca-46e2-a264-3415b03bf9f6","Type":"ContainerDied","Data":"4bb395f5b9139973a816fae620426f7dda77e6c3d5d780fb82ded1877e99d245"} Nov 03 19:00:23.280245 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.280213 4885 generic.go:356] "Generic (PLEG): container finished" podID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerID="e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec" exitCode=0 Nov 03 19:00:23.280245 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.280237 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" event={"ID":"b9795628-59b5-44cb-baf3-d93ea91ce738","Type":"ContainerDied","Data":"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec"} Nov 03 19:00:23.280432 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.280283 4885 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" Nov 03 19:00:23.280432 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.280290 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl" event={"ID":"b9795628-59b5-44cb-baf3-d93ea91ce738","Type":"ContainerDied","Data":"4549bc42b08cbfbd5b4b24169b82722970bd6cd38ef74336b7c366622465eff8"} Nov 03 19:00:23.287493 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.287241 4885 scope.go:122] "RemoveContainer" containerID="f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734" Nov 03 19:00:23.287841 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:00:23.287587 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734\": container with ID starting with f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734 not found: ID does not exist" containerID="f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734" Nov 03 19:00:23.287841 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.287643 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734"} err="failed to get container status \"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734\": rpc error: code = NotFound desc = could not find container \"f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734\": container with ID starting with f55463f443b5b90a0f219e3c62e9c19a295d75bed7d0df13464f62ff816d2734 not found: ID does not exist" Nov 03 19:00:23.287841 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.287662 4885 scope.go:122] "RemoveContainer" containerID="b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639" Nov 03 19:00:23.306357 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.306340 4885 scope.go:122] "RemoveContainer" containerID="b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639" Nov 03 19:00:23.306646 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:00:23.306620 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639\": container with ID starting with b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639 not found: ID does not exist" containerID="b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639" Nov 03 19:00:23.306700 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.306658 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639"} err="failed to get container status \"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639\": rpc error: code = NotFound desc = could not find container \"b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639\": container with ID starting with b716892a314c2de48e2e4940acc5a4b06eed33074a5a46852408f7054f711639 not found: ID does not exist" Nov 03 19:00:23.306700 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.306684 4885 scope.go:122] "RemoveContainer" containerID="e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec" Nov 03 19:00:23.326265 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.326236 4885 scope.go:122] "RemoveContainer" containerID="e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec" Nov 03 19:00:23.332488 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:00:23.332463 4885 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec\": container with ID starting with e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec not found: ID does not exist" containerID="e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec" Nov 03 19:00:23.332624 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.332496 4885 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec"} err="failed to get container status \"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec\": rpc error: code = NotFound desc = could not find container \"e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec\": container with ID starting with e4d3d6295029142d18bea22f11489f4e12747603384218efe5aa8552549f2cec not found: ID does not exist" Nov 03 19:00:23.567949 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.567874 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 19:00:23.571727 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.571698 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-927sl"] Nov 03 19:00:23.660931 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.660897 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" path="/var/lib/kubelet/pods/b9795628-59b5-44cb-baf3-d93ea91ce738/volumes" Nov 03 19:00:23.683755 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.683726 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 19:00:23.688162 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.688135 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-csqjk"] Nov 03 19:00:23.803222 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.803186 4885 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 19:00:23.806807 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:23.806782 4885 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-zg2g2"] Nov 03 19:00:25.661244 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:25.661211 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" path="/var/lib/kubelet/pods/69a32526-b77b-4ca2-872b-f5c928ddd077/volumes" Nov 03 19:00:25.661628 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:00:25.661607 4885 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" path="/var/lib/kubelet/pods/90b428a3-01ca-46e2-a264-3415b03bf9f6/volumes" Nov 03 19:00:48.605293 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:00:48.605255 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:01:18.604606 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:01:18.604561 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:01:48.605336 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:01:48.605288 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:02:00.448814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:00.448779 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:02:18.605511 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:02:18.605470 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:02:33.799863 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.799830 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2"] Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800147 4885 cpu_manager.go:414] "containerMap: removing container" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800158 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800172 4885 cpu_manager.go:414] "containerMap: removing container" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800177 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800187 4885 cpu_manager.go:414] "containerMap: removing container" podUID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" containerName="maas-api" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800191 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" containerName="maas-api" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800201 4885 cpu_manager.go:414] "containerMap: removing container" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800206 4885 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800258 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="90b428a3-01ca-46e2-a264-3415b03bf9f6" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800266 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9795628-59b5-44cb-baf3-d93ea91ce738" containerName="istio-proxy" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800272 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0b4e67-9c00-4dd7-b30a-a65c4101856b" containerName="maas-api" Nov 03 19:02:33.800358 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.800277 4885 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a32526-b77b-4ca2-872b-f5c928ddd077" containerName="istio-proxy" Nov 03 19:02:33.803270 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.803250 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.848944 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.848910 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlw5x\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-kube-api-access-tlw5x\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849103 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.848984 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849103 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849055 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849204 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849110 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849242 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849174 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849242 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849232 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849342 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849264 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849342 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849299 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.849445 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.849431 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.856797 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.856770 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-t4h2g\"" Nov 03 19:02:33.949849 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949814 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949874 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlw5x\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-kube-api-access-tlw5x\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949899 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949923 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949954 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.949984 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950019 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950009 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950360 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950047 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950360 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950099 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950360 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950314 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950516 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950461 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950516 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950483 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950597 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950557 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.950702 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.950681 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.952401 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.952384 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:33.952653 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:33.952637 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:34.004095 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.004046 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2"] Nov 03 19:02:34.012840 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.012811 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:34.012961 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.012921 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlw5x\" (UniqueName: \"kubernetes.io/projected/dec8f363-ccf4-46da-b521-a6a4ecfed2e0-kube-api-access-tlw5x\") pod \"maas-default-gateway-openshift-default-58c6585447-lxfr2\" (UID: \"dec8f363-ccf4-46da-b521-a6a4ecfed2e0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:34.116093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.115991 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:34.531823 ip-10-0-129-76 kubenswrapper[4885]: W1103 19:02:34.531791 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec8f363_ccf4_46da_b521_a6a4ecfed2e0.slice/crio-85a04de79d9662c5bfea233a080427d4e392b5747629567b3f2696a8ddb50217 WatchSource:0}: Error finding container 85a04de79d9662c5bfea233a080427d4e392b5747629567b3f2696a8ddb50217: Status 404 returned error can't find the container with id 85a04de79d9662c5bfea233a080427d4e392b5747629567b3f2696a8ddb50217 Nov 03 19:02:34.533760 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.533732 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2"] Nov 03 19:02:34.533962 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.533931 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:34.534030 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.533991 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:34.534030 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.534019 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:34.813809 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.813730 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" event={"ID":"dec8f363-ccf4-46da-b521-a6a4ecfed2e0","Type":"ContainerStarted","Data":"5e0696c3347c4a5bef28b9970dfd160bddfce36b340fb3fdaa07eec3a9eaae63"} Nov 03 19:02:34.813809 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.813788 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" event={"ID":"dec8f363-ccf4-46da-b521-a6a4ecfed2e0","Type":"ContainerStarted","Data":"85a04de79d9662c5bfea233a080427d4e392b5747629567b3f2696a8ddb50217"} Nov 03 19:02:34.937473 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:34.937425 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podStartSLOduration=1.9374113419999999 podStartE2EDuration="1.937411342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:34.937346946 +0000 UTC m=+6297.392035985" watchObservedRunningTime="2026-11-03 19:02:34.937411342 +0000 UTC m=+6297.392100364" Nov 03 19:02:35.125134 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:35.124877 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:35.125134 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:35.125006 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:35.125134 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:35.125047 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:36.120091 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:36.119442 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:36.120091 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:36.119520 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:37.125093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:37.124299 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:37.125093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:37.124353 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:38.126542 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:38.126500 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:38.127204 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:38.127165 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:39.125147 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:39.124449 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:39.125147 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:39.124511 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:40.119470 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:40.119428 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:40.120085 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:40.120032 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:41.117247 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:41.117204 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:41.117430 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:41.117278 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:42.117122 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:42.117085 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 19:02:42.117525 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:42.117147 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 19:02:43.929936 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:43.929900 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:43.930475 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:43.930345 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:43.931293 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:43.931269 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" Nov 03 19:02:48.607733 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:02:48.607694 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:02:48.908040 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.908005 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d"] Nov 03 19:02:48.911171 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.911154 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.965642 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965606 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.965830 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965647 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.965830 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965733 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.965830 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965751 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.966012 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965834 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxz2x\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-kube-api-access-mxz2x\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.966012 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965898 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.966012 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965940 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.966012 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.965966 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:48.966174 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:48.966031 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dff2d5-08cb-4428-9954-f659cac0dab1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.007935 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.007901 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d"] Nov 03 19:02:49.067413 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067372 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067413 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067416 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067670 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067432 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067670 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067559 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dff2d5-08cb-4428-9954-f659cac0dab1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067670 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067635 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067670 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067667 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067886 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067710 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067886 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067739 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067886 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067780 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxz2x\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-kube-api-access-mxz2x\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067886 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067836 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.067886 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067858 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.068194 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.067989 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.068255 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.068226 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49dff2d5-08cb-4428-9954-f659cac0dab1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.068493 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.068432 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.070058 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.070036 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.070340 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.070322 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.131197 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.131166 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.132134 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.132060 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxz2x\" (UniqueName: \"kubernetes.io/projected/49dff2d5-08cb-4428-9954-f659cac0dab1-kube-api-access-mxz2x\") pod \"maas-default-gateway-openshift-default-58c6585447-vx22d\" (UID: \"49dff2d5-08cb-4428-9954-f659cac0dab1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.223250 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.223166 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:49.639975 ip-10-0-129-76 kubenswrapper[4885]: W1103 19:02:49.639945 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49dff2d5_08cb_4428_9954_f659cac0dab1.slice/crio-d530959f68d6039bb6a9ea9f7f4f17cf093819ed3e18c780e478b2f95418814f WatchSource:0}: Error finding container d530959f68d6039bb6a9ea9f7f4f17cf093819ed3e18c780e478b2f95418814f: Status 404 returned error can't find the container with id d530959f68d6039bb6a9ea9f7f4f17cf093819ed3e18c780e478b2f95418814f Nov 03 19:02:49.640984 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.640962 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d"] Nov 03 19:02:49.642389 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.642356 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:49.642456 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.642413 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:49.642456 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.642439 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:02:49.891488 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.891398 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" event={"ID":"49dff2d5-08cb-4428-9954-f659cac0dab1","Type":"ContainerStarted","Data":"d7ada1b70f8c31934cd61191b4ef9a6fdadcd042f063dfac854525ac1b85ea37"} Nov 03 19:02:49.891488 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:49.891462 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" event={"ID":"49dff2d5-08cb-4428-9954-f659cac0dab1","Type":"ContainerStarted","Data":"d530959f68d6039bb6a9ea9f7f4f17cf093819ed3e18c780e478b2f95418814f"} Nov 03 19:02:50.014407 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:50.014353 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podStartSLOduration=2.01433867 podStartE2EDuration="2.01433867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:50.013663293 +0000 UTC m=+6312.468352331" watchObservedRunningTime="2026-11-03 19:02:50.01433867 +0000 UTC m=+6312.469027695" Nov 03 19:02:50.225234 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:50.225094 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:50.225873 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:50.225541 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:50.225873 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:50.225590 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:51.226506 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:51.226463 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:51.226930 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:51.226520 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:52.226552 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:52.226508 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:52.226964 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:52.226585 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:53.226554 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:53.226511 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:53.226744 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:53.226572 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:54.226574 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:54.226532 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:54.226763 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:54.226601 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:55.226484 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:55.226444 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:55.226640 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:55.226498 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:56.223724 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:56.223679 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:56.224275 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:56.223734 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:57.224332 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:57.224291 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 03 19:02:57.224828 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:57.224342 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 03 19:02:58.994340 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:58.994285 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:58.994849 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:58.994600 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:02:58.995517 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:02:58.995492 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" Nov 03 19:03:18.605356 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:03:18.605320 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:03:48.604778 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:03:48.604737 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:03:52.662154 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:03:52.662105 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:04:18.603531 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:04:18.603487 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:04:19.692770 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.692738 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn"] Nov 03 19:04:19.695987 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.695967 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766434 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766393 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766611 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766443 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766611 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766528 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766611 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766571 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766611 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766596 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766755 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766623 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766755 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766698 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9094206-30b8-48a8-a812-7ef3261ae80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766755 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766723 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxs9\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-kube-api-access-qjxs9\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.766851 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.766773 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.792847 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.792815 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn"] Nov 03 19:04:19.867205 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867161 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867400 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867220 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867400 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867330 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867400 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867373 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867635 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867433 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9094206-30b8-48a8-a812-7ef3261ae80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867635 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867482 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxs9\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-kube-api-access-qjxs9\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867635 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867514 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867635 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867545 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867848 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867647 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867848 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867693 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867961 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867869 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.867961 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867941 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.868042 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.867978 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.868275 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.868256 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9094206-30b8-48a8-a812-7ef3261ae80d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.870023 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.870002 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.870130 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.870046 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.930699 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.930662 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:19.930892 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:19.930869 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxs9\" (UniqueName: \"kubernetes.io/projected/f9094206-30b8-48a8-a812-7ef3261ae80d-kube-api-access-qjxs9\") pod \"maas-default-gateway-openshift-default-58c6585447-wc8gn\" (UID: \"f9094206-30b8-48a8-a812-7ef3261ae80d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:20.008149 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:20.008051 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:20.425005 ip-10-0-129-76 kubenswrapper[4885]: W1103 19:04:20.424974 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9094206_30b8_48a8_a812_7ef3261ae80d.slice/crio-8bbf33ca8aa0c7a37c19e4418bf5fec45bff1f7fb1ce36ef449bca3068b77cec WatchSource:0}: Error finding container 8bbf33ca8aa0c7a37c19e4418bf5fec45bff1f7fb1ce36ef449bca3068b77cec: Status 404 returned error can't find the container with id 8bbf33ca8aa0c7a37c19e4418bf5fec45bff1f7fb1ce36ef449bca3068b77cec Nov 03 19:04:20.425983 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:20.425958 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn"] Nov 03 19:04:20.427044 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:20.427014 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:04:20.427132 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:20.427097 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:04:20.427178 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:20.427149 4885 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 19:04:21.293218 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:21.293187 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" event={"ID":"f9094206-30b8-48a8-a812-7ef3261ae80d","Type":"ContainerStarted","Data":"08faeeb8569d11f5ea4a464a182d45be04ce3a431d96e03b6d30f066e98ad21e"} Nov 03 19:04:21.293611 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:21.293237 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" event={"ID":"f9094206-30b8-48a8-a812-7ef3261ae80d","Type":"ContainerStarted","Data":"8bbf33ca8aa0c7a37c19e4418bf5fec45bff1f7fb1ce36ef449bca3068b77cec"} Nov 03 19:04:21.468759 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:21.468711 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podStartSLOduration=2.46869636 podStartE2EDuration="2.46869636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:21.467525023 +0000 UTC m=+6403.922214048" watchObservedRunningTime="2026-11-03 19:04:21.46869636 +0000 UTC m=+6403.923385385" Nov 03 19:04:22.017097 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:22.015711 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:22.017097 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:22.015860 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:22.017097 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:22.015945 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:23.022093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:23.018525 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:23.022093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:23.018589 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:24.011092 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:24.009469 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:24.011092 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:24.009529 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:25.013558 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:25.013516 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:25.014195 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:25.014159 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:26.019093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:26.013425 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:26.019093 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:26.013501 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:27.008844 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:27.008806 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:27.009031 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:27.008858 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:28.008507 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:28.008464 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 03 19:04:28.008904 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:28.008544 4885 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 03 19:04:29.537226 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:29.537136 4885 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:29.537668 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:29.537631 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:29.538485 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:04:29.538464 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" Nov 03 19:04:48.604117 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:04:48.604055 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:05:18.605076 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:05:18.605028 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:05:41.739503 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:05:41.739464 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:05:48.610910 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:05:48.610869 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:06:18.604956 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:06:18.604918 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:06:48.605285 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:06:48.605247 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:07:18.605620 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:07:18.605575 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:07:36.740150 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:07:36.740052 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:07:48.606200 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:07:48.606165 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:07:56.750828 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:07:56.750790 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:07:59.980225 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:07:59.980175 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:07:59.980854 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:07:59.980817 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:00.036197 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:00.036158 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:00.036509 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:00.036474 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:00.597231 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:00.597192 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:08:00.597429 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:00.597283 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:08:18.605672 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:08:18.605634 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:08:34.466282 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:34.466245 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:08:47.138774 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:08:47.138739 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:08:48.604994 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:08:48.604952 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:09:10.943814 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:09:10.943734 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:09:18.607387 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:09:18.607347 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:09:23.568783 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:09:23.568741 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:09:48.604788 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:09:48.604752 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:09:57.227088 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:09:57.227021 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:10:07.848579 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:10:07.848544 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:10:15.526377 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:10:15.526346 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:10:18.683329 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:10:18.683284 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:10:28.628673 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:10:28.628600 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:10:48.605112 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:10:48.605047 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:10:52.224844 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:10:52.224811 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:11:02.956274 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:02.956242 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:11:11.459316 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:11.459284 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:11:18.605419 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:11:18.605378 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:11:29.980283 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:29.980220 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:11:29.980903 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:29.980306 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:11:29.995235 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:29.995143 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:11:29.995392 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:29.995269 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:11:33.728440 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:33.728403 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:11:48.605516 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:11:48.605474 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:11:56.222897 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:56.222861 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:11:59.977309 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:59.977194 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:11:59.977829 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:11:59.977312 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:12:00.039092 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:12:00.038393 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:12:00.039092 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:12:00.038471 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:12:18.605817 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:12:18.605708 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:12:48.606095 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:12:48.606039 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:13:18.604638 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:13:18.604602 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:13:44.977300 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:44.977244 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-lxfr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:13:44.977830 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:44.977328 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-lxfr2" podUID="dec8f363-ccf4-46da-b521-a6a4ecfed2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:45.037196 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:45.037158 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-vx22d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:13:45.037547 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:45.037514 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-vx22d" podUID="49dff2d5-08cb-4428-9954-f659cac0dab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:45.586306 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:45.586255 4885 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-wc8gn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 19:13:45.586544 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:45.586328 4885 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-wc8gn" podUID="f9094206-30b8-48a8-a812-7ef3261ae80d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 19:13:48.604539 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:13:48.604499 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:13:54.243621 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:13:54.243587 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sdh8h"] Nov 03 19:14:18.607673 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:14:18.607631 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:14:26.125385 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:26.125352 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-r7rw5_19e0a745-46bf-4cb6-a26b-bc8c643aa60d/manager/0.log" Nov 03 19:14:26.449513 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:26.449436 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xj552_6b2b23ec-041e-4b09-bc89-fa5e2399b5fc/manager/0.log" Nov 03 19:14:27.027623 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:27.027585 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hqg4v_7fbbf0df-7541-4486-bc3c-861bea91ef4e/manager/0.log" Nov 03 19:14:30.433927 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:30.433890 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-669c98d779-lhckj_3ad86a4a-6768-4e95-a0c0-eebfebb52d21/maas-api/0.log" Nov 03 19:14:37.711900 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:37.711868 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4lcxm_b9b78432-2478-417e-9226-2d987a705dc5/manager/0.log" Nov 03 19:14:38.027411 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:38.027328 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xbdkc_7e51fea8-9d06-422e-9176-70c40477d2ab/manager/0.log" Nov 03 19:14:39.574889 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:39.574853 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gqqlv_47ac64e0-e994-4c7d-93ea-344eb970eb9d/manager/0.log" Nov 03 19:14:40.580691 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:40.580660 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sdh8h_2003487e-5c5d-4661-bfdb-b9b72e729cb6/limitador/0.log" Nov 03 19:14:42.273181 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:42.273146 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rvgtt_91117bda-a060-42e8-a82f-466590963e91/discovery/0.log" Nov 03 19:14:44.358541 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:44.358516 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-lxfr2_dec8f363-ccf4-46da-b521-a6a4ecfed2e0/istio-proxy/0.log" Nov 03 19:14:44.929119 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:44.929092 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-vx22d_49dff2d5-08cb-4428-9954-f659cac0dab1/istio-proxy/0.log" Nov 03 19:14:45.225781 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:14:45.225706 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-wc8gn_f9094206-30b8-48a8-a812-7ef3261ae80d/istio-proxy/0.log" Nov 03 19:14:48.604143 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:14:48.604105 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:15:11.001057 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:11.000981 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z526b_9fb96298-6581-4c6b-aa6f-4a6ce4a07eeb/global-pull-secret-syncer/0.log" Nov 03 19:15:14.123742 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:14.123709 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xlbvs_54863910-20c1-44fe-b7f7-f2a39321a9eb/konnectivity-agent/0.log" Nov 03 19:15:14.680111 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:14.680061 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-76.us-west-2.compute.internal_31fb60344eae41808f24a7b8d8670170/haproxy/0.log" Nov 03 19:15:18.604922 ip-10-0-129-76 kubenswrapper[4885]: E1103 19:15:18.604874 4885 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-89c8l" containerName="extractor" Nov 03 19:15:18.827119 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:18.827090 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-r7rw5_19e0a745-46bf-4cb6-a26b-bc8c643aa60d/manager/0.log" Nov 03 19:15:19.082332 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:19.082296 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xj552_6b2b23ec-041e-4b09-bc89-fa5e2399b5fc/manager/0.log" Nov 03 19:15:20.102320 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.102293 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-4lcxm_b9b78432-2478-417e-9226-2d987a705dc5/manager/0.log" Nov 03 19:15:20.170747 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.170711 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hqg4v_7fbbf0df-7541-4486-bc3c-861bea91ef4e/manager/0.log" Nov 03 19:15:20.250177 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.250137 4885 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b"] Nov 03 19:15:20.253609 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.253591 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.303755 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.303713 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqm4b\" (UniqueName: \"kubernetes.io/projected/a98749fd-5c87-4ea9-b804-4d7597a34850-kube-api-access-dqm4b\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.303951 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.303823 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-proc\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.303951 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.303856 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-sys\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.303951 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.303926 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-podres\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.304094 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.304008 4885 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-lib-modules\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.335523 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.335490 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b"] Nov 03 19:15:20.397720 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.397670 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45cfb\"/\"kube-root-ca.crt\"" Nov 03 19:15:20.405299 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405268 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-podres\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405482 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405331 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-lib-modules\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405482 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405369 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqm4b\" (UniqueName: \"kubernetes.io/projected/a98749fd-5c87-4ea9-b804-4d7597a34850-kube-api-access-dqm4b\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405482 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405428 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-proc\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405482 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405431 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-podres\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405482 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405453 4885 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-sys\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405766 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405507 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-lib-modules\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405766 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405530 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-proc\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.405766 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.405615 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98749fd-5c87-4ea9-b804-4d7597a34850-sys\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.423988 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.423968 4885 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-45cfb\"/\"openshift-service-ca.crt\"" Nov 03 19:15:20.424122 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.423993 4885 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-45cfb\"/\"default-dockercfg-chcxq\"" Nov 03 19:15:20.468326 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.468300 4885 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqm4b\" (UniqueName: \"kubernetes.io/projected/a98749fd-5c87-4ea9-b804-4d7597a34850-kube-api-access-dqm4b\") pod \"perf-node-gather-daemonset-9lk9b\" (UID: \"a98749fd-5c87-4ea9-b804-4d7597a34850\") " pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.566635 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.566608 4885 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:20.755416 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.755333 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-xbdkc_7e51fea8-9d06-422e-9176-70c40477d2ab/manager/0.log" Nov 03 19:15:20.930645 ip-10-0-129-76 kubenswrapper[4885]: W1103 19:15:20.930612 4885 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda98749fd_5c87_4ea9_b804_4d7597a34850.slice/crio-075abf765d6482131c38fe36c8550bf56b38efb56be078380fb048f346203bba WatchSource:0}: Error finding container 075abf765d6482131c38fe36c8550bf56b38efb56be078380fb048f346203bba: Status 404 returned error can't find the container with id 075abf765d6482131c38fe36c8550bf56b38efb56be078380fb048f346203bba Nov 03 19:15:20.932328 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:20.932300 4885 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b"] Nov 03 19:15:21.438964 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:21.438927 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" event={"ID":"a98749fd-5c87-4ea9-b804-4d7597a34850","Type":"ContainerStarted","Data":"5ba0498a04beae86d2a10b437b3cd6b648719993e1cdfebd17b0c4f143a9259f"} Nov 03 19:15:21.439440 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:21.438977 4885 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" event={"ID":"a98749fd-5c87-4ea9-b804-4d7597a34850","Type":"ContainerStarted","Data":"075abf765d6482131c38fe36c8550bf56b38efb56be078380fb048f346203bba"} Nov 03 19:15:21.439440 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:21.439012 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:21.567654 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:21.567605 4885 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" podStartSLOduration=1.567590831 podStartE2EDuration="1.567590831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:21.566765308 +0000 UTC m=+7064.021454333" watchObservedRunningTime="2026-11-03 19:15:21.567590831 +0000 UTC m=+7064.022279856" Nov 03 19:15:21.893362 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:21.893331 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-gqqlv_47ac64e0-e994-4c7d-93ea-344eb970eb9d/manager/0.log" Nov 03 19:15:22.263125 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:22.263021 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sdh8h_2003487e-5c5d-4661-bfdb-b9b72e729cb6/limitador/0.log" Nov 03 19:15:25.247827 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:25.247793 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-r7rw5_19e0a745-46bf-4cb6-a26b-bc8c643aa60d/manager/0.log" Nov 03 19:15:25.262496 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:25.262464 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/alertmanager/0.log" Nov 03 19:15:25.499346 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:25.499266 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xj552_6b2b23ec-041e-4b09-bc89-fa5e2399b5fc/manager/0.log" Nov 03 19:15:25.863727 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:25.863653 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/config-reloader/0.log" Nov 03 19:15:26.198925 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:26.198891 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hqg4v_7fbbf0df-7541-4486-bc3c-861bea91ef4e/manager/0.log" Nov 03 19:15:26.528095 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:26.528000 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/kube-rbac-proxy-web/0.log" Nov 03 19:15:26.924871 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:26.924840 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/kube-rbac-proxy/0.log" Nov 03 19:15:27.322202 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:27.322125 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/kube-rbac-proxy-metric/0.log" Nov 03 19:15:27.454662 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:27.454638 4885 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-45cfb/perf-node-gather-daemonset-9lk9b" Nov 03 19:15:27.719485 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:27.719459 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/prom-label-proxy/0.log" Nov 03 19:15:28.118343 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:28.118315 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f2e737e2-92b0-41e7-bdd5-826b84824632/init-config-reloader/0.log" Nov 03 19:15:31.009979 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:31.009950 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mxk9_b7b90722-afde-44ca-88ac-a73e3b3c61aa/node-exporter/0.log" Nov 03 19:15:31.237675 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:31.237648 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-r7rw5_19e0a745-46bf-4cb6-a26b-bc8c643aa60d/manager/0.log" Nov 03 19:15:31.405740 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:31.405712 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mxk9_b7b90722-afde-44ca-88ac-a73e3b3c61aa/kube-rbac-proxy/0.log" Nov 03 19:15:31.493236 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:31.493201 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-xj552_6b2b23ec-041e-4b09-bc89-fa5e2399b5fc/manager/0.log" Nov 03 19:15:32.007313 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:32.007283 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8mxk9_b7b90722-afde-44ca-88ac-a73e3b3c61aa/init-textfile/0.log" Nov 03 19:15:32.071463 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:32.071435 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-hqg4v_7fbbf0df-7541-4486-bc3c-861bea91ef4e/manager/0.log" Nov 03 19:15:40.292714 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:40.292682 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rvgtt_91117bda-a060-42e8-a82f-466590963e91/discovery/0.log" Nov 03 19:15:41.946005 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:41.945977 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-lxfr2_dec8f363-ccf4-46da-b521-a6a4ecfed2e0/istio-proxy/0.log" Nov 03 19:15:42.370778 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:42.370747 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-vx22d_49dff2d5-08cb-4428-9954-f659cac0dab1/istio-proxy/0.log" Nov 03 19:15:42.580461 ip-10-0-129-76 kubenswrapper[4885]: I1103 19:15:42.580434 4885 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-wc8gn_f9094206-30b8-48a8-a812-7ef3261ae80d/istio-proxy/0.log"