Oct 30 19:18:41.779319 ip-10-0-132-153 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:18:42.221851 ip-10-0-132-153 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:18:42.221851 ip-10-0-132-153 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:18:42.221851 ip-10-0-132-153 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:18:42.221851 ip-10-0-132-153 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227773 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227865 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227869 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227872 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227876 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.227869 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227878 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227881 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227884 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227887 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227890 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227892 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227895 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227900 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227905 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227909 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227912 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227915 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227918 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227921 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227923 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227926 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227929 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227932 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227934 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.228118 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227941 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227944 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227947 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227950 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227952 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227955 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227958 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227960 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227963 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227966 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227969 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227971 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227992 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.227996 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228001 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228005 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228008 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228011 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228013 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228016 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.228570 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228019 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228022 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228024 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228027 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228030 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228032 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228036 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228040 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228042 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228045 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228048 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228050 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228053 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228056 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228058 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228061 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228063 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228066 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228069 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228071 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.229058 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228074 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228076 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228079 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228082 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228084 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228087 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228090 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228093 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228096 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228099 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228101 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228104 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228107 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228110 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228113 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228116 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228118 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228121 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228124 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.229524 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228128 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228132 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228135 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228138 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228141 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228143 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228146 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228150 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228153 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228156 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228159 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228161 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228164 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228167 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228170 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228172 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228175 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228178 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228180 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.229994 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228183 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228186 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228189 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.228193 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229037 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229046 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229049 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229052 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229055 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229057 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229060 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229063 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229066 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229068 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229072 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229074 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229077 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229079 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229082 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229085 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.230446 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229087 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229090 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229092 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229095 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229098 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229100 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229103 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229106 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229108 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229111 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229113 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229116 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229120 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229123 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229126 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229128 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229131 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229134 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229136 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229139 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.231042 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229142 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229144 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229147 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229149 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229152 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229154 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229157 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229160 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229163 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229165 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229168 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229170 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229173 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229175 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229178 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229181 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229183 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229185 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229188 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.231523 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229191 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229193 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229196 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229198 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229201 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229205 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229208 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229211 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229213 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229216 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229219 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229221 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229224 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229226 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229229 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229231 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229234 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229236 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229239 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229241 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.231996 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229244 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229248 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229250 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229253 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229256 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229258 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229261 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229263 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229266 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229271 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229275 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229278 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229280 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229283 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229286 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229288 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229291 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229295 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.232499 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229299 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229302 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229306 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229308 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229311 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229314 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229317 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229319 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229322 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229324 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229328 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229331 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229334 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229846 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229851 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229855 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229859 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229861 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229864 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229867 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.232923 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229870 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229874 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229878 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229882 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229885 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229888 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229890 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229893 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229896 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229900 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229904 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229907 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229910 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229913 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229915 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229918 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229920 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229923 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229925 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.233416 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229928 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229930 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229933 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229935 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229938 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229940 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229943 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229945 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229948 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229951 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229957 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229959 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229962 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229964 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229967 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229969 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229972 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229991 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229994 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.229998 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.233883 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230001 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230004 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230007 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230010 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230013 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230015 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230018 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230021 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230023 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230026 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230028 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230031 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230034 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230036 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230038 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230041 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230043 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230046 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230048 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.234363 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230051 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230053 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230056 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230061 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230063 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230066 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230069 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230071 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230074 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230077 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230079 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230082 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230085 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230087 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230090 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230093 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230095 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230098 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230100 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230103 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.234799 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230106 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230108 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230111 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230113 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230116 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230118 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230121 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230123 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230126 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230129 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230131 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230134 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230136 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230139 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230141 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230145 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230148 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230151 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230154 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230157 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.235317 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230159 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230258 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230265 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230272 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230276 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230280 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230284 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230293 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230297 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230300 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230303 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230307 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230311 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230314 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230318 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230321 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230324 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230327 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230330 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230334 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230337 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230341 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230344 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230347 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:18:42.235793 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230351 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230354 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230358 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230364 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230367 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230371 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230374 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230377 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230380 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230386 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230389 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230392 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230395 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230398 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230401 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230405 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230409 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230411 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230415 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230418 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230422 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230425 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230428 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230431 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230434 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:18:42.236370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230437 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230440 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230443 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230446 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230449 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230452 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230457 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230460 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230463 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230466 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230470 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230474 2575 flags.go:64] FLAG: --help="false" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230477 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230481 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230484 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230487 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230491 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230495 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230498 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230501 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230504 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230507 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230510 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:18:42.237152 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230513 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230516 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230519 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230522 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230525 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230528 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230531 2575 flags.go:64] FLAG: --lock-file="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230534 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230537 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230540 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230545 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230548 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230552 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230555 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230558 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230561 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230564 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230567 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230571 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230576 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230580 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230583 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230586 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230590 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230593 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:18:42.237712 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230597 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230600 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230603 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230611 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230614 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230617 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230620 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230623 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230626 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230629 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230632 2575 flags.go:64] FLAG: --port="10250" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230635 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230638 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03d09ec56d3790992" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230641 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230644 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230647 2575 flags.go:64] FLAG: --register-node="true" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230650 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230654 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230657 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230660 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230663 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230667 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230670 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230673 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230676 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230679 2575 flags.go:64] FLAG: --runonce="false" Oct 30 19:18:42.238343 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230683 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230687 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230690 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230693 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230696 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230699 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230703 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230706 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230709 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230712 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230715 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230718 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230721 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230724 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230727 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230732 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230735 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230738 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230743 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230746 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230749 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230752 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230755 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230758 2575 flags.go:64] FLAG: --v="2" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230762 2575 flags.go:64] FLAG: --version="false" Oct 30 19:18:42.238963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230766 2575 flags.go:64] FLAG: --vmodule="" Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230770 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.230774 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230904 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230908 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230910 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230913 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230917 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230920 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230923 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230926 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230928 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230931 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230936 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230939 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230942 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230944 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230947 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230949 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230952 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.239574 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230955 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230957 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230960 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230962 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230965 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230968 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230985 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230989 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230991 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230994 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.230997 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231000 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231002 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231005 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231015 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231018 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231021 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231024 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231028 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.240071 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231031 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231034 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231037 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231039 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231042 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231046 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231050 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231052 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231056 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231058 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231061 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231064 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231066 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231069 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231072 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231074 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231077 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231080 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231083 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231085 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.240593 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231088 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231090 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231093 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231096 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231099 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231101 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231104 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231107 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231109 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231112 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231114 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231117 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231120 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231123 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231125 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231128 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231130 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231134 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231136 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.241102 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231143 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231146 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231148 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231151 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231154 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231156 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231159 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231163 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231172 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231176 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231178 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231181 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231184 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231186 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231189 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231192 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231194 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231197 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231200 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231202 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.241550 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231205 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231209 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231213 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231216 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231219 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231222 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231225 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231227 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231230 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231233 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.231260 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231302 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231307 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231316 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231319 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.242040 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231322 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231324 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231327 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231330 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231332 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231335 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231338 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231340 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231343 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231345 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231348 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231350 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231353 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231356 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231358 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231361 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231363 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231366 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231368 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231370 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.242398 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231373 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231376 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231378 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231381 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231383 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231386 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231389 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231391 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231396 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231399 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231402 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231405 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231407 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231410 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231412 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231415 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231418 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231420 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.242864 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231423 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231425 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231428 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231430 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231433 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231435 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231438 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231440 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231443 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231446 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231448 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231451 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231453 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231456 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231459 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231461 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231464 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231466 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231469 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.243342 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231471 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231474 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231477 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231479 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231482 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231484 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231487 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231490 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231492 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231495 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231497 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231500 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231503 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231507 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231511 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231514 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231517 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231520 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231522 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.243785 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231528 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231531 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231533 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231536 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231539 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231551 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231555 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231561 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231565 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231569 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231572 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231575 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231578 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231582 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231584 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231588 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231591 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231593 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.244321 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231596 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231599 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231602 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231604 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231607 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.231609 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.238352 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.238370 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.238386 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.238391 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238460 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238465 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238468 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238472 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238475 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238477 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238480 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238483 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238485 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.244814 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238488 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238491 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238493 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238496 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238498 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238501 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238504 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238507 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238509 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238512 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238515 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238518 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238521 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238524 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238526 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238529 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238532 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238535 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238537 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238540 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.245362 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238543 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238546 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238549 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238551 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238554 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238557 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238559 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238562 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238565 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238567 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238570 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238573 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238575 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238578 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238581 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238583 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238586 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238588 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238591 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238594 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.245899 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238597 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238599 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238602 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238605 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238607 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238610 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238613 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238616 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238619 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238622 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238624 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238627 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238630 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238633 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238648 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238653 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238658 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238662 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238665 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.246516 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238668 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238671 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238674 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238676 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238679 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238682 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238684 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238687 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238691 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238694 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238697 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238700 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238703 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238705 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238708 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238710 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238714 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238716 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238719 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.247124 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238721 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238724 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238727 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238730 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238735 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238738 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238740 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238746 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238749 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238752 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238755 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238757 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238760 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238762 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238765 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238768 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238770 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.247604 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238773 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.238800 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238959 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238964 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238968 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238987 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238993 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238996 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.238999 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239002 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239004 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239007 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239010 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239012 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239015 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.248133 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239018 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239021 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239023 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239026 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239028 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239031 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239033 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239036 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239040 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239043 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239046 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239048 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239051 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239054 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239057 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239059 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239062 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239064 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239067 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239070 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.248667 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239072 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239076 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239079 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239083 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239086 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239089 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239092 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239094 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239097 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239099 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239102 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239105 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239108 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239110 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239113 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239115 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239118 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239121 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239124 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.249296 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239126 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239129 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239132 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239134 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239137 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239139 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239142 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239145 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239148 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239150 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239153 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239155 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239158 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239160 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239163 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239165 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239168 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239171 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239173 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239176 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.249885 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239179 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239181 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239184 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239187 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239190 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239193 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239195 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239198 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239200 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239203 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239206 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239208 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239210 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239213 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239215 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239218 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239221 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239223 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239226 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239229 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.250384 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239232 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239234 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239237 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239239 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239242 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239244 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239247 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239249 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239252 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239254 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239257 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239259 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239262 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239265 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.239288 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239318 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:18:42.250854 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239322 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239327 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239331 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239334 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239337 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239339 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239342 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239345 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239347 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239350 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239353 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239355 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239358 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239361 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239366 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239369 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239371 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239374 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:18:42.251258 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239376 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239379 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239382 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239384 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239387 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239390 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239392 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239395 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239397 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239400 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239404 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239407 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239409 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239412 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239415 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239419 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239422 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239425 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239429 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:18:42.251687 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239432 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239435 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239437 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239440 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239442 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239445 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239447 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239450 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239453 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239458 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239460 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239463 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239466 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239469 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239471 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239474 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239477 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239479 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239482 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239484 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:18:42.252196 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239487 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239489 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239492 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239496 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239499 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239501 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239504 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239507 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239509 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239512 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239514 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239517 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239520 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239522 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239527 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239530 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239533 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239535 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239538 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:18:42.252685 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239540 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239545 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239547 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239550 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239552 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239555 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239558 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239560 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239563 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239565 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239568 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239570 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239573 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239576 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239579 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239583 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239585 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239588 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239591 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239594 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:18:42.253183 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239596 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239599 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239601 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239604 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239606 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239609 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239612 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:42.239614 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.239666 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.241658 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.242521 2575 server.go:1014] "Starting client certificate rotation" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.242620 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.243443 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.244528 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:18:42.253698 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.250716 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:18:42.261790 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.261772 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:18:42.265258 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.265243 2575 log.go:25] "Validated CRI v1 image API" Oct 30 19:18:42.266161 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.266133 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:18:42.268663 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.268640 2575 fs.go:135] Filesystem UUIDs: map[0cf2dfd4-42d6-4be2-ab48-4f729b1e9faf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe513efd-079e-424e-a28b-6958259b3539:/dev/nvme0n1p3] Oct 30 19:18:42.268740 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.268660 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:18:42.274122 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.273998 2575 manager.go:217] Machine: {Timestamp:2026-10-30 19:18:42.272677598 +0000 UTC m=+0.361047218 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101244 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fb9645611140446ba102e536df658 SystemUUID:ec2fb964-5611-1404-46ba-102e536df658 BootID:60900f46-05e4-474d-824d-d24ba23b8e7e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:a4:e1:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:a4:e1:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:cc:13:99:42:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:18:42.274242 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.274134 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:18:42.274293 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.274254 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:18:42.275435 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275410 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:18:42.275642 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275436 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:18:42.275743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275647 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:18:42.275743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275660 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:18:42.275743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275708 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:18:42.275743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.275736 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:18:42.276520 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.276499 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:18:42.276577 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.276507 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:18:42.295725 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.295701 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lvpjd" Oct 30 19:18:42.300598 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300582 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:18:42.300741 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300731 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:18:42.300777 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300769 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:18:42.300804 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300784 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:18:42.300804 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300796 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:18:42.300862 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300807 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:18:42.300862 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.300825 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:18:42.301291 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.301271 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:18:42.304483 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.304357 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:18:42.304820 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.304783 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lvpjd" Oct 30 19:18:42.309143 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.309124 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:18:42.310149 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310129 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:18:42.310230 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310178 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:18:42.310813 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310795 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:18:42.310813 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310816 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:18:42.310914 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310825 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:18:42.310914 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310834 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:18:42.310914 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310842 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:18:42.310914 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.310851 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:18:42.311326 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311315 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:18:42.311374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311330 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:18:42.311374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311343 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:18:42.311374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311352 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:18:42.311374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311367 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:18:42.311525 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.311381 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:18:42.312475 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.312458 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:18:42.312546 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.312480 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:18:42.316677 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.316660 2575 server.go:1263] "Started kubelet" Oct 30 19:18:42.316854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.316789 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:18:42.316908 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.316866 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:18:42.317201 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.316882 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:18:42.317407 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.317389 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:42.317542 ip-10-0-132-153 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:18:42.318003 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.317537 2575 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-132-153.us-east-2.compute.internal" not found Oct 30 19:18:42.319191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.319177 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:18:42.319590 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.319563 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:18:42.322109 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.322089 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:18:42.323563 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.323545 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:18:42.323642 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.323567 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:18:42.323754 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.323727 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-153.us-east-2.compute.internal\" not found" Oct 30 19:18:42.324326 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.324310 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:18:42.324639 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.324624 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:18:42.324716 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.324313 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:18:42.324716 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.324490 2575 factory.go:55] Registering systemd factory Oct 30 19:18:42.324805 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.324726 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:18:42.325062 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325040 2575 factory.go:153] Registering CRI-O factory Oct 30 19:18:42.325062 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325058 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 19:18:42.325213 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325113 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:18:42.325213 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325136 2575 factory.go:103] Registering Raw factory Oct 30 19:18:42.325213 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325151 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 19:18:42.325323 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325256 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:18:42.325323 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.325266 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:18:42.326018 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.326001 2575 manager.go:319] Starting recovery of all containers Oct 30 19:18:42.327434 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.327328 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:42.329111 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.329085 2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-153.us-east-2.compute.internal\" not found" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.330937 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.330911 2575 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-132-153.us-east-2.compute.internal" not found Oct 30 19:18:42.337159 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.337141 2575 manager.go:324] Recovery completed Oct 30 19:18:42.338750 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.338726 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 19:18:42.341695 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.341683 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:18:42.343547 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.343528 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:18:42.343608 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.343560 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:18:42.343608 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.343574 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:18:42.344108 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.344097 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:18:42.344146 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.344106 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:18:42.344146 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.344123 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:18:42.344209 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.344166 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:18:42.345613 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.345598 2575 policy_none.go:50] "Start" Oct 30 19:18:42.345691 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.345619 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:18:42.345691 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.345632 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:18:42.346284 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.346267 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:18:42.346355 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.346300 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:18:42.346355 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.346315 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:18:42.347212 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.347200 2575 policy_none.go:44] "Start" Oct 30 19:18:42.378569 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.378535 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:18:42.382479 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.382458 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 19:18:42.382593 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.382502 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:18:42.382593 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.382515 2575 server.go:87] "Starting device plugin registration server" Oct 30 19:18:42.382819 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.382804 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:18:42.382890 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.382824 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:18:42.382943 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.382933 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:18:42.383039 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.383022 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:18:42.383039 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.383037 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:18:42.383762 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.383580 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:18:42.383849 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.383803 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-153.us-east-2.compute.internal\" not found" Oct 30 19:18:42.384225 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.384211 2575 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-132-153.us-east-2.compute.internal" not found Oct 30 19:18:42.388491 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.388476 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:42.470514 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.470471 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:18:42.472450 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.471723 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:18:42.472450 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.471742 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:18:42.472450 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.471766 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:18:42.472450 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.471806 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:18:42.476506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.476490 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:42.483578 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.483564 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:18:42.484476 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.484456 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:18:42.484534 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.484495 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:18:42.484534 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.484509 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:18:42.484534 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.484534 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.494028 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.494014 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.494083 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:42.494036 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-153.us-east-2.compute.internal\": node \"ip-10-0-132-153.us-east-2.compute.internal\" not found" Oct 30 19:18:42.572816 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.572781 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal"] Oct 30 19:18:42.575161 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.575144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.576893 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.576873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.603625 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.603605 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.606528 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.606506 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.617771 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.617751 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:18:42.620507 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.620494 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:18:42.626825 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.626808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.626887 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.626834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.626887 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.626854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727478 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727408 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727478 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727595 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727595 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727595 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727560 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:42.727595 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:42.727589 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:44.556557 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.556527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:44.559627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.559609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:18:44.877423 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.877396 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:18:44.877949 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.877576 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:18:44.877949 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.877597 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:18:44.877949 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.877597 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:18:44.937078 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.937049 2575 apiserver.go:52] "Watching apiserver" Oct 30 19:18:44.944251 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.944214 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:13:43 +0000 UTC" deadline="2028-05-09 13:29:51.725654413 +0000 UTC" Oct 30 19:18:44.944251 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.944245 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13362h11m6.781413396s" Oct 30 19:18:44.944537 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.944524 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:18:44.944807 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.944791 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-h9j28","kube-system/global-pull-secret-syncer-n84k8","openshift-ovn-kubernetes/ovnkube-node-wv29b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8","openshift-cluster-node-tuning-operator/tuned-x8548","openshift-dns/node-resolver-6kvp2","openshift-image-registry/node-ca-cxnr9","openshift-multus/multus-additional-cni-plugins-cfdq9","openshift-multus/multus-h2bcc","kube-system/konnectivity-agent-h9jxz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-dmtjr","openshift-network-operator/iptables-alerter-xlfrt","kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal"] Oct 30 19:18:44.947099 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.947083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:44.947193 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:44.947176 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:44.950745 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.949273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:44.950745 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:44.949364 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:44.951643 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.951623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.953809 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.953787 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.955942 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.955915 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:18:44.956046 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.955964 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:18:44.956046 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.955986 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:18:44.956046 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.955984 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:18:44.956046 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.955997 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.956169 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.956048 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6rfl7\"" Oct 30 19:18:44.956169 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.956009 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:18:44.956169 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.956100 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:18:44.956970 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.956956 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:18:44.957111 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.957100 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:18:44.957205 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.957194 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vd8dh\"" Oct 30 19:18:44.957265 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.957255 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:18:44.957961 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.957946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:44.959081 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.959067 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:18:44.959225 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.959213 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:18:44.960044 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.960029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:44.962112 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.962095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.964203 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.964183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2bcc" Oct 30 19:18:44.966123 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.966207 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966144 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:44.966207 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966366 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966366 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:44.966366 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966366 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966592 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz44v\" (UniqueName: \"kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.966935 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.966992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7w2\" (UniqueName: \"kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967133 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srnf\" (UniqueName: \"kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:44.967334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967347 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkh7w\" (UniqueName: \"kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfcd\" (UniqueName: \"kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.967833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.967962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bxs\" (UniqueName: \"kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.968005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:44.968358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.968062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:44.968964 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.968676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:44.968964 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:44.968737 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:44.970900 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.970883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:44.973035 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.973017 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:18:44.996096 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:44.996069 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4ndr2" Oct 30 19:18:45.004802 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.004782 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4ndr2" Oct 30 19:18:45.056484 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056457 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:18:45.056484 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056468 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.056752 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056486 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.056752 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056496 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:18:45.056961 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056941 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:18:45.057028 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.056984 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:18:45.057028 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057028 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.057134 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057044 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:18:45.057211 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057190 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:18:45.057211 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057202 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:18:45.057348 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057218 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ctbj8\"" Oct 30 19:18:45.057780 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057765 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:18:45.057879 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057866 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:18:45.057936 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057907 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:18:45.057936 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.057911 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.058330 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058315 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vj9wr\"" Oct 30 19:18:45.058408 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058363 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n5j86\"" Oct 30 19:18:45.058408 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058372 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:18:45.058408 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058386 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dk8pw\"" Oct 30 19:18:45.058408 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058405 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:18:45.058579 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058400 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t67x5\"" Oct 30 19:18:45.058579 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058419 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vhzsf\"" Oct 30 19:18:45.058579 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058407 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhqgv\"" Oct 30 19:18:45.058579 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.058489 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:18:45.061348 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.061334 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068354 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068410 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.068506 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068486 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068749 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068766 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068851 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.069002 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.068968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cw9\" (UniqueName: \"kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069170 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069199 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz44v\" (UniqueName: \"kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069212 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.069762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069326 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069353 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069401 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069465 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069521 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069539 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069649 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.070566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069662 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7w2\" (UniqueName: \"kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srnf\" (UniqueName: \"kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.069970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070052 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.071341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070099 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070281 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkh7w\" (UniqueName: \"kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070394 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070637 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.072064 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070853 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070896 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfv2\" (UniqueName: \"kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.070995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071051 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.071062 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfcd\" (UniqueName: \"kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.072580 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.071105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:45.571091919 +0000 UTC m=+2.023994159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071136 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrpc\" (UniqueName: \"kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071369 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bxs\" (UniqueName: \"kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071517 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.073107 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071862 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.073694 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071871 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.073694 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.071886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.073820 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.073798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073899 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.073864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.073899 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.073877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.074745 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.074726 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:18:45.074846 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.074752 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:18:45.074846 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.074766 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:45.074846 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.074814 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:18:45.574798422 +0000 UTC m=+2.027700660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:45.076908 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.076886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz44v\" (UniqueName: \"kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.078637 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.078613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7w2\" (UniqueName: \"kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.079252 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.079234 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srnf\" (UniqueName: \"kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.079544 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.079528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfcd\" (UniqueName: \"kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.080564 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.080538 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkh7w\" (UniqueName: \"kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.083540 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.083519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bxs\" (UniqueName: \"kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.172081 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172176 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172176 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172228 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172235 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.172249 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172383 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172397 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfv2\" (UniqueName: \"kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.172418 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:45.672384085 +0000 UTC m=+2.125286320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:45.172545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172526 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrpc\" (UniqueName: \"kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cw9\" (UniqueName: \"kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.172967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173572 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.173236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173572 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.173300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173572 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.173350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.173572 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.173399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.174001 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.173960 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9876e1c179ef0e136f2b6103bc70569.slice/crio-8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da WatchSource:0}: Error finding container 8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da: Status 404 returned error can't find the container with id 8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da Oct 30 19:18:45.174209 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.174187 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.174250 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.174196 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.174298 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.174268 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.174396 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.174378 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.175059 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.175039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.180218 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.180201 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:18:45.183818 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.183799 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfv2\" (UniqueName: \"kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.184136 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.184121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrpc\" (UniqueName: \"kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.203643 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.203616 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae4921b5faf0892072f47ac728ceb26.slice/crio-df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7 WatchSource:0}: Error finding container df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7: Status 404 returned error can't find the container with id df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7 Oct 30 19:18:45.207862 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.207839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cw9\" (UniqueName: \"kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.287832 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.287805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:18:45.297664 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.297639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:18:45.297901 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.297880 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92253fd_8219_46e9_b7c6_a4ccebbd7dd4.slice/crio-b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9 WatchSource:0}: Error finding container b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9: Status 404 returned error can't find the container with id b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9 Oct 30 19:18:45.306558 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.306527 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01332dde_ec0a_41fd_abfc_f73fab46b857.slice/crio-8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34 WatchSource:0}: Error finding container 8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34: Status 404 returned error can't find the container with id 8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34 Oct 30 19:18:45.308998 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.308973 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:18:45.317648 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.317626 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5dcce8_8409_491b_8428_eecdaf1ea2c0.slice/crio-7638478a510006d43d08ccb873190093b60f6ba7528ee735698e4de4b4291f72 WatchSource:0}: Error finding container 7638478a510006d43d08ccb873190093b60f6ba7528ee735698e4de4b4291f72: Status 404 returned error can't find the container with id 7638478a510006d43d08ccb873190093b60f6ba7528ee735698e4de4b4291f72 Oct 30 19:18:45.317722 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.317691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:18:45.325410 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.325378 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb207c2c_2bf8_4263_b349_aa2f5be7b0fb.slice/crio-6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0 WatchSource:0}: Error finding container 6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0: Status 404 returned error can't find the container with id 6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0 Oct 30 19:18:45.334528 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.334510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:18:45.343693 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.343670 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dd1ff4_96b4_430d_b31c_ef3a77fe9f18.slice/crio-c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195 WatchSource:0}: Error finding container c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195: Status 404 returned error can't find the container with id c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195 Oct 30 19:18:45.361567 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.361548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:18:45.367799 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.367631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h2bcc" Oct 30 19:18:45.372457 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.372410 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d167a43_62ce_4e3b_98ad_a095c3276ea8.slice/crio-08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5 WatchSource:0}: Error finding container 08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5: Status 404 returned error can't find the container with id 08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5 Oct 30 19:18:45.376305 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.376279 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcea2b7_e349_4e35_b71a_35ad85f61c54.slice/crio-573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62 WatchSource:0}: Error finding container 573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62: Status 404 returned error can't find the container with id 573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62 Oct 30 19:18:45.400032 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.400012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:45.403914 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.403889 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:18:45.409797 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.409770 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8b6a1e_6af3_4e39_9f3b_7b05da2fbf23.slice/crio-ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636 WatchSource:0}: Error finding container ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636: Status 404 returned error can't find the container with id ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636 Oct 30 19:18:45.411760 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:18:45.411710 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96d731e_41cf_442f_adfc_3d36a9908b31.slice/crio-d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9 WatchSource:0}: Error finding container d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9: Status 404 returned error can't find the container with id d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9 Oct 30 19:18:45.575903 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.575873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:45.575903 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.575912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:45.576111 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576041 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:18:45.576111 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576058 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:18:45.576111 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576067 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:45.576111 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576109 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:18:46.576097926 +0000 UTC m=+3.029000159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:45.576247 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576040 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:45.576247 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.576178 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:46.576162647 +0000 UTC m=+3.029064870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:45.676626 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.676533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:45.676779 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.676708 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:45.676827 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:45.676791 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:46.676770891 +0000 UTC m=+3.129673127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:45.781253 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.781211 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:45.868737 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:45.868707 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:46.008903 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.008729 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:13:44 +0000 UTC" deadline="2028-06-01 14:03:31.546977191 +0000 UTC" Oct 30 19:18:46.008903 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.008791 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13914h44m45.538189927s" Oct 30 19:18:46.110191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.109566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:46.110191 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.109700 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:46.110191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.109733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:46.110191 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.109801 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:46.135746 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.135660 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kvp2" event={"ID":"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb","Type":"ContainerStarted","Data":"6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0"} Oct 30 19:18:46.151999 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.151924 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x8548" event={"ID":"4e5dcce8-8409-491b-8428-eecdaf1ea2c0","Type":"ContainerStarted","Data":"7638478a510006d43d08ccb873190093b60f6ba7528ee735698e4de4b4291f72"} Oct 30 19:18:46.174833 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.174786 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" event={"ID":"01332dde-ec0a-41fd-abfc-f73fab46b857","Type":"ContainerStarted","Data":"8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34"} Oct 30 19:18:46.184701 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.184664 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"1ae4921b5faf0892072f47ac728ceb26","Type":"ContainerStarted","Data":"df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7"} Oct 30 19:18:46.191411 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.191365 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxnr9" event={"ID":"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18","Type":"ContainerStarted","Data":"c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195"} Oct 30 19:18:46.204765 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.204724 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9"} Oct 30 19:18:46.222176 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.222135 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerStarted","Data":"8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da"} Oct 30 19:18:46.231241 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.231188 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlfrt" event={"ID":"c96d731e-41cf-442f-adfc-3d36a9908b31","Type":"ContainerStarted","Data":"d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9"} Oct 30 19:18:46.250494 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.250439 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h9jxz" event={"ID":"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23","Type":"ContainerStarted","Data":"ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636"} Oct 30 19:18:46.278089 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.278003 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2bcc" event={"ID":"8dcea2b7-e349-4e35-b71a-35ad85f61c54","Type":"ContainerStarted","Data":"573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62"} Oct 30 19:18:46.292579 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.292508 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerStarted","Data":"08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5"} Oct 30 19:18:46.303082 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.303050 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:18:46.588268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.588186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:46.588268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.588248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:46.588585 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588456 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:18:46.588585 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588477 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:18:46.588585 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588491 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:46.588585 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588546 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:18:48.588528678 +0000 UTC m=+5.041430912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:46.588799 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588640 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:46.588799 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.588675 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:48.58866329 +0000 UTC m=+5.041565529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:46.689334 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:46.689300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:46.689624 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.689421 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:46.689624 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:46.689501 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:48.689484419 +0000 UTC m=+5.142386656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:47.010620 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:47.009965 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:13:44 +0000 UTC" deadline="2028-06-26 05:42:37.807608354 +0000 UTC" Oct 30 19:18:47.010620 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:47.010008 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14506h23m50.79760565s" Oct 30 19:18:47.107874 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:47.107842 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:47.108073 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:47.107987 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:48.111190 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:48.111157 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:48.111590 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.111289 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:48.112650 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:48.111641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:48.112650 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.111750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:48.603774 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:48.603737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:48.603944 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:48.603798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:48.604026 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.603968 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:18:48.604026 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.603988 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:18:48.604026 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.604002 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:48.604175 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.604064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:18:52.604042929 +0000 UTC m=+9.056945168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:48.604492 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.604472 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:48.604589 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.604528 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:52.604513743 +0000 UTC m=+9.057415984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:48.705079 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:48.704424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:48.705079 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.704654 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:48.705079 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:48.704718 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:18:52.704700104 +0000 UTC m=+9.157602339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:49.108468 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:49.108379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:49.108627 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:49.108550 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:49.340071 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:49.339806 2575 generic.go:356] "Generic (PLEG): container finished" podID="e9876e1c179ef0e136f2b6103bc70569" containerID="b5494c5f72d38ba5f60770882adca52ead2dc012c2af4463de94c2466190a458" exitCode=0 Oct 30 19:18:49.340764 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:49.340253 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerDied","Data":"b5494c5f72d38ba5f60770882adca52ead2dc012c2af4463de94c2466190a458"} Oct 30 19:18:49.348660 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:49.348628 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerStarted","Data":"250c1b246fa4a50e6581de0715bc06808f2c4936f2739a6584e5636fb0b510f3"} Oct 30 19:18:50.107560 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.107531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:50.107736 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:50.107657 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:50.107736 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.107531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:50.107856 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:50.107745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:50.369147 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.369104 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="250c1b246fa4a50e6581de0715bc06808f2c4936f2739a6584e5636fb0b510f3" exitCode=0 Oct 30 19:18:50.369537 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.369189 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"250c1b246fa4a50e6581de0715bc06808f2c4936f2739a6584e5636fb0b510f3"} Oct 30 19:18:50.384910 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.384282 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"1ae4921b5faf0892072f47ac728ceb26","Type":"ContainerStarted","Data":"621039d322aa10514228f85c9d5e30f49d52b9e8361c54e2c8afc067d96ee5cb"} Oct 30 19:18:50.388172 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.387548 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerStarted","Data":"5f059f4047ae026e8af63827b24dd3c92c8e3b96d257a129e809b3a40e6563fa"} Oct 30 19:18:50.421711 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:50.421110 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" podStartSLOduration=6.421096985 podStartE2EDuration="6.421096985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:50.407815003 +0000 UTC m=+6.860717248" watchObservedRunningTime="2026-10-30 19:18:50.421096985 +0000 UTC m=+6.873999224" Oct 30 19:18:51.107612 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:51.107461 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:51.107764 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:51.107670 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:51.392418 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:51.392340 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h9jxz" event={"ID":"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23","Type":"ContainerStarted","Data":"a2cb97240518196461a0eecb2e059a269e5b8e993013bbce45a73de6f6692b1c"} Oct 30 19:18:51.397394 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:51.397364 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxnr9" event={"ID":"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18","Type":"ContainerStarted","Data":"7b59ccacbb42dfec1a59454971f05d0bc3a9d1b22ce82ee760d9fd0fd216dc17"} Oct 30 19:18:51.413226 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:51.413159 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" podStartSLOduration=7.413138123 podStartE2EDuration="7.413138123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:50.421695195 +0000 UTC m=+6.874597435" watchObservedRunningTime="2026-10-30 19:18:51.413138123 +0000 UTC m=+7.866040366" Oct 30 19:18:51.415390 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:51.415292 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h9jxz" podStartSLOduration=2.687345795 podStartE2EDuration="7.415234144s" totalImagesPullingTime="4.727888349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:51.413707257 +0000 UTC m=+7.866609491" watchObservedRunningTime="2026-10-30 19:18:51.415234144 +0000 UTC m=+7.868136382" Oct 30 19:18:52.111388 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.110614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:52.111388 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.110752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:52.111388 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.111170 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:52.111388 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.111270 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:52.405672 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.405593 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" event={"ID":"01332dde-ec0a-41fd-abfc-f73fab46b857","Type":"ContainerStarted","Data":"51eb80ef01bdb58fc33312ceb47d5caade57836d87cbada1bdab4ee67f4dcc00"} Oct 30 19:18:52.646371 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.646338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:52.646566 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.646392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:52.646618 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646567 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:18:52.646618 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646590 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:18:52.646618 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646603 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:52.646732 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646655 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:19:00.646638354 +0000 UTC m=+17.099540591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:18:52.646732 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646726 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:52.646817 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.646767 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:00.646754742 +0000 UTC m=+17.099656979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:18:52.747399 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.747317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:52.747603 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.747562 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:52.747673 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:52.747638 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:00.747617775 +0000 UTC m=+17.200520011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:18:52.903273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.903188 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:52.903724 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.903696 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:52.919411 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:52.919293 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cxnr9" podStartSLOduration=3.348821884 podStartE2EDuration="8.919275777s" totalImagesPullingTime="5.570453893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:51.4394859 +0000 UTC m=+7.892388140" watchObservedRunningTime="2026-10-30 19:18:52.919275777 +0000 UTC m=+9.372178020" Oct 30 19:18:53.107918 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:53.107852 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:53.108060 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:53.108012 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:53.345358 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:53.345278 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:18:53.415128 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:53.415042 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" event={"ID":"01332dde-ec0a-41fd-abfc-f73fab46b857","Type":"ContainerStarted","Data":"3ee1392847b334967ea6ce2126a9d91f48ee4b49d9cb7a91c48ec63d438a83c1"} Oct 30 19:18:54.027706 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.026908 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:18:53.345291608Z","UUID":"ef1ea399-40bd-4ee9-b878-4d1f0e13d45a","Handler":null,"Name":"","Endpoint":""} Oct 30 19:18:54.029678 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.029549 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:18:54.029678 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.029580 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:18:54.126963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.126928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:54.127122 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:54.127082 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:54.127615 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.127583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:54.129842 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:54.129804 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:54.418832 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.418710 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h2bcc" event={"ID":"8dcea2b7-e349-4e35-b71a-35ad85f61c54","Type":"ContainerStarted","Data":"de331839a5e63d825e86d5d85b3321adbaff12042faa642ca3fad5271d98ec51"} Oct 30 19:18:54.421341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.421319 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:18:54.422034 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.421964 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kvp2" event={"ID":"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb","Type":"ContainerStarted","Data":"f0103a6820dd70bc25c5158afaaa2ff8f80427455e4abfacd7898aef77964a86"} Oct 30 19:18:54.437294 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.437246 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h2bcc" podStartSLOduration=2.121408411 podStartE2EDuration="10.437232007s" totalImagesPullingTime="8.315823596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:54.437144905 +0000 UTC m=+10.890047148" watchObservedRunningTime="2026-10-30 19:18:54.437232007 +0000 UTC m=+10.890134251" Oct 30 19:18:54.455244 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:54.455187 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6kvp2" podStartSLOduration=2.361176625 podStartE2EDuration="10.455167586s" totalImagesPullingTime="8.093990961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:54.454490454 +0000 UTC m=+10.907392697" watchObservedRunningTime="2026-10-30 19:18:54.455167586 +0000 UTC m=+10.908069831" Oct 30 19:18:55.108287 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.108183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:55.108422 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:55.108323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:55.425483 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.425350 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="2552b9828dd4e2d158c5d0e56df30be0b213568c53122e0dd4a4e652561eefa7" exitCode=0 Oct 30 19:18:55.425483 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.425462 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"2552b9828dd4e2d158c5d0e56df30be0b213568c53122e0dd4a4e652561eefa7"} Oct 30 19:18:55.430521 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.430494 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" event={"ID":"01332dde-ec0a-41fd-abfc-f73fab46b857","Type":"ContainerStarted","Data":"244718585ace8bee68b3e25bc31d58d0f0a699b6a736f90ae94086eabeebd051"} Oct 30 19:18:55.464716 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.464028 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" podStartSLOduration=2.71845613 podStartE2EDuration="11.464010395s" totalImagesPullingTime="8.745554265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:55.463682181 +0000 UTC m=+11.916584424" watchObservedRunningTime="2026-10-30 19:18:55.464010395 +0000 UTC m=+11.916912638" Oct 30 19:18:55.571021 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:55.570997 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxnr9_a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/node-ca/0.log" Oct 30 19:18:56.110726 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.108415 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:56.110726 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.108432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:56.110726 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:56.108801 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:56.110726 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:56.108867 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:56.435100 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.435061 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="cddbe5d123ce1cbfff79f70e6744beb9eecbecfa54b3de72bc3831fa2c83ab7a" exitCode=0 Oct 30 19:18:56.435538 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.435151 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"cddbe5d123ce1cbfff79f70e6744beb9eecbecfa54b3de72bc3831fa2c83ab7a"} Oct 30 19:18:56.437090 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.436723 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x8548" event={"ID":"4e5dcce8-8409-491b-8428-eecdaf1ea2c0","Type":"ContainerStarted","Data":"f918c34af567615efdb82835fdf2132b5316937a92d97c7c68462da7d85533f1"} Oct 30 19:18:56.438307 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.438279 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlfrt" event={"ID":"c96d731e-41cf-442f-adfc-3d36a9908b31","Type":"ContainerStarted","Data":"6823af00b538e37799b01be6db3fe2a2e473b1f21dbc1287383ad98fad236ebc"} Oct 30 19:18:56.469120 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.469075 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xlfrt" podStartSLOduration=3.830480263 podStartE2EDuration="12.469062562s" totalImagesPullingTime="8.638582299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:56.468862992 +0000 UTC m=+12.921765238" watchObservedRunningTime="2026-10-30 19:18:56.469062562 +0000 UTC m=+12.921964803" Oct 30 19:18:56.485623 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.485581 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x8548" podStartSLOduration=2.159569709 podStartE2EDuration="12.485568734s" totalImagesPullingTime="10.325999025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:56.485126316 +0000 UTC m=+12.938028556" watchObservedRunningTime="2026-10-30 19:18:56.485568734 +0000 UTC m=+12.938470976" Oct 30 19:18:56.787241 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.787158 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:56.787395 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.787305 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:18:56.787765 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:56.787745 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:18:57.108133 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:57.108058 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:57.108298 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:57.108196 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:18:58.110578 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:58.110543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:18:58.111315 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:58.110554 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:18:58.111315 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:58.110691 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:18:58.111315 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:58.110749 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:18:58.446213 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:58.446123 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="822d5e16e30f43cde1728faca4ba885e87f27c8ca4f0d4fdb11ea5f4c273d2d0" exitCode=0 Oct 30 19:18:58.446213 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:58.446186 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"822d5e16e30f43cde1728faca4ba885e87f27c8ca4f0d4fdb11ea5f4c273d2d0"} Oct 30 19:18:59.108024 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:18:59.107981 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:18:59.108212 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:18:59.108114 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:00.107968 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:00.107936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:00.108511 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:00.107940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:00.108511 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.108054 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:00.108511 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.108136 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:00.713373 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:00.713128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:00.713373 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713333 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:19:00.713373 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713356 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:19:00.713373 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713371 2575 projected.go:196] Error preparing data for projected volume kube-api-access-vjz6k for pod openshift-network-diagnostics/network-check-target-h9j28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:19:00.713373 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:00.713374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:00.713634 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713480 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k podName:dc2ecc8f-2f83-4ead-a40a-f3d487ba622b nodeName:}" failed. No retries permitted until 2026-10-30 19:19:16.713412116 +0000 UTC m=+33.166314339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vjz6k" (UniqueName: "kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k") pod "network-check-target-h9j28" (UID: "dc2ecc8f-2f83-4ead-a40a-f3d487ba622b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:19:00.713634 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713481 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:00.713634 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.713529 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:16.713516249 +0000 UTC m=+33.166418484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:00.814508 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:00.814486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:00.814637 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.814618 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:00.814703 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:00.814692 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:16.814671159 +0000 UTC m=+33.267573385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:01.108327 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.108084 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:01.108731 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:01.108371 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:01.455640 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.455567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:01.455912 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.455891 2575 generic.go:356] "Generic (PLEG): container finished" podID="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" containerID="06f0505eb1de34387f98b0f34f42b0ab369201452abc80e281b0e48720d93903" exitCode=1 Oct 30 19:19:01.455998 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.455977 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"733873d86796881f4c54517d4e96cd4b07783de87cbe5e1fce96333269f5327f"} Oct 30 19:19:01.456066 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.456053 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"a206e1f8f78acf27846dc0f99512e43597cc98353d9865b2a7b4c2cac666eb9d"} Oct 30 19:19:01.456119 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.456087 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"19f7031b867530d38360c5f6f38129a87a533b28fd6f489f0760ec70fc9b50a6"} Oct 30 19:19:01.456164 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.456124 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"6e265538bab5cbdf86630b0a572e1854665317fe1c46ff5d48350a21e5626bd6"} Oct 30 19:19:01.456164 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.456153 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerDied","Data":"06f0505eb1de34387f98b0f34f42b0ab369201452abc80e281b0e48720d93903"} Oct 30 19:19:01.456246 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.456186 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"accfb508343179e2429427fd38b61c8be0209f70258511b65865e6b3ccd56044"} Oct 30 19:19:01.458767 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.458746 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="e1e7194435d553a015cea68457760d28ba9abbee444bda27a9713a79c5e515be" exitCode=0 Oct 30 19:19:01.458865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:01.458784 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"e1e7194435d553a015cea68457760d28ba9abbee444bda27a9713a79c5e515be"} Oct 30 19:19:02.108231 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:02.108190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:02.108231 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:02.108214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:02.108965 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:02.108310 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:02.108965 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:02.108426 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:02.463773 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:02.463682 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="817238c930acd8196dd00a44f41f7b8856968b9e2c25b10f0b9af070e154126c" exitCode=0 Oct 30 19:19:02.463773 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:02.463731 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerDied","Data":"817238c930acd8196dd00a44f41f7b8856968b9e2c25b10f0b9af070e154126c"} Oct 30 19:19:03.107647 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:03.107612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:03.107817 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:03.107752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:03.469273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:03.469237 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" event={"ID":"9d167a43-62ce-4e3b-98ad-a095c3276ea8","Type":"ContainerStarted","Data":"83a2d1857f118ab5bda165f910b2f974e44b079264ae9f0161256202d309c378"} Oct 30 19:19:03.472379 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:03.472359 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:03.472730 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:03.472710 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"3c72b8a6d5834e3028ac46c1ad48cb7cacdf64b9c63e4bc545d5ad24e64284bf"} Oct 30 19:19:03.494749 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:03.494703 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cfdq9" podStartSLOduration=8.724736693 podStartE2EDuration="19.494690134s" totalImagesPullingTime="10.769953441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:03.494327887 +0000 UTC m=+19.947230131" watchObservedRunningTime="2026-10-30 19:19:03.494690134 +0000 UTC m=+19.947592376" Oct 30 19:19:04.108340 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:04.108309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:04.108522 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:04.108459 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:04.108522 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:04.108488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:04.108599 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:04.108543 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:05.108126 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.108094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:05.108741 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:05.108205 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:05.480109 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.480029 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:05.480379 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.480353 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"2c32aba4fedfb9ce60ca61f1576f234cda8209dfa0334eccdf73558efd52a267"} Oct 30 19:19:05.480701 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.480681 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:05.480773 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.480757 2575 scope.go:122] "RemoveContainer" containerID="06f0505eb1de34387f98b0f34f42b0ab369201452abc80e281b0e48720d93903" Oct 30 19:19:05.480829 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.480761 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:05.497344 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.497325 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:05.497669 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:05.497653 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:06.107811 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.107771 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:06.108014 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:06.107880 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:06.108014 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.107946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:06.108134 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:06.108037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:06.485881 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.485810 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:06.486293 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.486136 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" event={"ID":"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4","Type":"ContainerStarted","Data":"acba38f11b49eb998db25161bb16fdb5ea50e95db8fac71873089686b11ebac5"} Oct 30 19:19:06.486293 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.486244 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:19:06.513755 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:06.513701 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" podStartSLOduration=7.325847736 podStartE2EDuration="22.513687765s" totalImagesPullingTime="15.187840029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:06.512214441 +0000 UTC m=+22.965116682" watchObservedRunningTime="2026-10-30 19:19:06.513687765 +0000 UTC m=+22.966590007" Oct 30 19:19:07.108341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:07.108310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:07.108580 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:07.108426 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:07.488069 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:07.488044 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:19:08.108367 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:08.108328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:08.108597 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:08.108328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:08.108597 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:08.108472 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:08.108597 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:08.108517 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:09.108374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:09.108339 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:09.108790 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:09.108474 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:10.108195 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.108165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:10.108374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.108214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:10.108374 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:10.108274 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:10.108374 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:10.108309 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:10.799599 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.799567 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n84k8"] Oct 30 19:19:10.799808 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.799673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:10.799808 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:10.799762 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:10.802744 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.802713 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9j28"] Oct 30 19:19:10.802949 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.802837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:10.802949 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:10.802936 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:10.803584 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.803551 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dmtjr"] Oct 30 19:19:10.803703 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:10.803646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:10.804026 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:10.803766 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:11.283983 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:11.283945 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:11.284724 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:11.284180 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:19:11.295310 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:11.295263 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" podUID="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:19:11.304037 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:11.304005 2575 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" podUID="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 19:19:12.108215 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:12.108178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:12.108386 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:12.108290 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:12.380990 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:12.380925 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kvp2_cb207c2c-2bf8-4263-b349-aa2f5be7b0fb/dns-node-resolver/0.log" Oct 30 19:19:13.107804 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:13.107768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:13.107804 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:13.107794 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:13.108068 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:13.107901 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:13.108068 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:13.108003 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:13.781466 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:13.781429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxnr9_a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/node-ca/0.log" Oct 30 19:19:14.111090 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:14.111016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:14.111228 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:14.111125 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9j28" podUID="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" Oct 30 19:19:15.108153 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.108121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:15.108153 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.108146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:15.108584 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:15.108235 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dmtjr" podUID="3b81ec2a-4636-47b1-93dd-c38322d944f7" Oct 30 19:19:15.108584 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:15.108347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n84k8" podUID="4738657b-e0c8-4a35-9893-85f99f3632b2" Oct 30 19:19:15.323010 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.322820 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeReady" Oct 30 19:19:15.323175 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.323105 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:19:15.372900 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.372865 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dxxbw"] Oct 30 19:19:15.399527 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.399502 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wrv7f"] Oct 30 19:19:15.399679 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.399613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.409254 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.409233 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:19:15.409386 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.409359 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktbkj\"" Oct 30 19:19:15.417460 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.417428 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxxbw"] Oct 30 19:19:15.417554 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.417487 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrv7f"] Oct 30 19:19:15.417554 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.417504 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fv898"] Oct 30 19:19:15.417630 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.417599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.429181 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.429158 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:19:15.429181 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.429158 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:19:15.429181 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.429174 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:19:15.432329 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.432314 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fv898"] Oct 30 19:19:15.432422 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.432412 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.506971 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.506946 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:19:15.507137 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.507100 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zxgfv\"" Oct 30 19:19:15.507137 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.507116 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:19:15.507137 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.507127 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gkcxf\"" Oct 30 19:19:15.517333 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.517394 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzwt\" (UniqueName: \"kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.517394 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.517500 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.517539 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.517539 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.517631 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs26\" (UniqueName: \"kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.517684 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.517747 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.517795 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.517831 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.517820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8h8\" (UniqueName: \"kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.529359 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.529338 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:19:15.529436 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.529405 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:19:15.529436 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.529412 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:19:15.618973 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.618895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.618973 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.618928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.618973 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.618945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.618973 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.618969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.619273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs26\" (UniqueName: \"kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.619273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.619273 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8h8\" (UniqueName: \"kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.619544 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619544 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzwt\" (UniqueName: \"kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619544 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619344 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619544 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.619731 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.619731 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.619712 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.620043 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.620018 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.623018 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.622998 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.623018 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.623014 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.623162 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.623011 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.629400 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.629378 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzwt\" (UniqueName: \"kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:15.629721 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.629705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs26\" (UniqueName: \"kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.629763 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.629729 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8h8\" (UniqueName: \"kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.739000 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.738944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:15.762874 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.762854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:15.795370 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:15.795324 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:16.053871 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.053836 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dxxbw"] Oct 30 19:19:16.058096 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:16.058067 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5364bcd9_a94f_4072_973c_bc3d9993d2bd.slice/crio-ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f WatchSource:0}: Error finding container ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f: Status 404 returned error can't find the container with id ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f Oct 30 19:19:16.078490 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.078467 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrv7f"] Oct 30 19:19:16.082098 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:16.082075 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f57a2e_06f1_4918_a6b3_e93802969d3d.slice/crio-860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce WatchSource:0}: Error finding container 860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce: Status 404 returned error can't find the container with id 860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce Oct 30 19:19:16.092866 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.092838 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fv898"] Oct 30 19:19:16.096365 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:16.096339 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa53f8d4_8928_4045_b84f_9d92c83c66e2.slice/crio-2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8 WatchSource:0}: Error finding container 2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8: Status 404 returned error can't find the container with id 2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8 Oct 30 19:19:16.108251 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.108228 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:16.112619 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.112598 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:19:16.112715 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.112607 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:19:16.112715 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.112655 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4vgss\"" Oct 30 19:19:16.508061 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.508029 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv898" event={"ID":"fa53f8d4-8928-4045-b84f-9d92c83c66e2","Type":"ContainerStarted","Data":"119bd06480f454e7e7223fc9b82c18bb50481c306f3a82929b68353b2db5573a"} Oct 30 19:19:16.508231 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.508087 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv898" event={"ID":"fa53f8d4-8928-4045-b84f-9d92c83c66e2","Type":"ContainerStarted","Data":"2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8"} Oct 30 19:19:16.509193 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.509152 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrv7f" event={"ID":"39f57a2e-06f1-4918-a6b3-e93802969d3d","Type":"ContainerStarted","Data":"860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce"} Oct 30 19:19:16.510112 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.510095 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxxbw" event={"ID":"5364bcd9-a94f-4072-973c-bc3d9993d2bd","Type":"ContainerStarted","Data":"ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f"} Oct 30 19:19:16.727823 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.727789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:16.728137 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.727929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:16.728137 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:16.728047 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:16.728248 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:16.728149 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret podName:4738657b-e0c8-4a35-9893-85f99f3632b2 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:48.728083942 +0000 UTC m=+65.180986162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret") pod "global-pull-secret-syncer-n84k8" (UID: "4738657b-e0c8-4a35-9893-85f99f3632b2") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:19:16.735259 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.735226 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:16.786079 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.785996 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:16.837684 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:16.837642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:16.837915 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:16.837889 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:16.837990 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:16.837982 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs podName:3b81ec2a-4636-47b1-93dd-c38322d944f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:48.837962343 +0000 UTC m=+65.290864568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs") pod "network-metrics-daemon-dmtjr" (UID: "3b81ec2a-4636-47b1-93dd-c38322d944f7") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:19:17.089321 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.089163 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9j28"] Oct 30 19:19:17.101863 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:17.101821 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ecc8f_2f83_4ead_a40a_f3d487ba622b.slice/crio-ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e WatchSource:0}: Error finding container ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e: Status 404 returned error can't find the container with id ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e Oct 30 19:19:17.108022 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.108000 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:17.108129 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.108031 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:17.113290 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.113267 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:19:17.113673 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.113319 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-t757m\"" Oct 30 19:19:17.113783 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.113767 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xs9gl\"" Oct 30 19:19:17.113856 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.113844 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:19:17.515576 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:17.515542 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9j28" event={"ID":"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b","Type":"ContainerStarted","Data":"ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e"} Oct 30 19:19:18.522486 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.522431 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv898" event={"ID":"fa53f8d4-8928-4045-b84f-9d92c83c66e2","Type":"ContainerStarted","Data":"ccd6bfa0b8b90f3c6a81f7e1f8c01f328853d5925e0e2037c0a4ebe3451ac966"} Oct 30 19:19:18.797753 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.797716 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq"] Oct 30 19:19:18.808936 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.808902 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz"] Oct 30 19:19:18.810770 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.810684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:18.822191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.821653 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:19:18.822191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.821993 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:19:18.822191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.822108 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:19:18.822191 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.822164 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:19:18.822778 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.822271 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9rkks\"" Oct 30 19:19:18.824166 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.824143 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:19:18.825125 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.824970 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq"] Oct 30 19:19:18.825125 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.825017 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p5rpc"] Oct 30 19:19:18.825584 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.825282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.843889 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.843815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.843889 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.843835 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz"] Oct 30 19:19:18.856295 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856274 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:19:18.856414 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856343 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j9nbc\"" Oct 30 19:19:18.856491 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856436 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:19:18.856582 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856567 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:19:18.856655 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856638 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7fl45\"" Oct 30 19:19:18.856708 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.856679 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:19:18.922681 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.922653 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:19:18.922854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.922733 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:19:18.955569 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.955743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:18.955743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.955743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:18.955743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.955743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955826 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglxr\" (UniqueName: \"kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5scm\" (UniqueName: \"kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.956004 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.955971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.956012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.956036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.956374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.956063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:18.956374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.956151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhlq\" (UniqueName: \"kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:18.956374 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:18.956232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.057714 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057669 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhlq\" (UniqueName: \"kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.057897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.057897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.057897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.057897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.057897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.058077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.058077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.058077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.057984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.058077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.058077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglxr\" (UniqueName: \"kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.058261 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5scm\" (UniqueName: \"kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.058261 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.058261 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.058261 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.058261 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.058551 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.058551 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.058551 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.058316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.059281 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.059253 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.059659 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.059632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.059961 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.059941 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.061431 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.061431 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061144 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.061431 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061264 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.061639 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061469 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.061787 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061758 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.061924 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.061902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.062560 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.062528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.064782 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.064757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.064875 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.064792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.065075 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.065052 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.066399 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.066370 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.066640 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.066616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.068666 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.068630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.069537 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.069518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhlq\" (UniqueName: \"kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.073821 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.073793 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglxr\" (UniqueName: \"kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.074327 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.074300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5scm\" (UniqueName: \"kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.180942 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.180645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:19.257327 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.257288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:19.274975 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.274943 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:19.522750 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.522717 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq"] Oct 30 19:19:19.526084 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.526058 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5rpc" event={"ID":"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0","Type":"ContainerStarted","Data":"d88c390d5522878eb56454aba7836af7b80cf176c8de256480b890590df8bc8b"} Oct 30 19:19:19.527734 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.527711 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxxbw" event={"ID":"5364bcd9-a94f-4072-973c-bc3d9993d2bd","Type":"ContainerStarted","Data":"84977f70b5682ff9fc3b76797bd95201201c69ec4dd0c1a849c40df4a4260e6c"} Oct 30 19:19:19.527829 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.527751 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dxxbw" event={"ID":"5364bcd9-a94f-4072-973c-bc3d9993d2bd","Type":"ContainerStarted","Data":"5251a93ad9ebe8a95f4d6b3d0d77bd293d236116815052df79e7a76f8be2c27f"} Oct 30 19:19:19.527938 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.527924 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:19.529858 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:19.529835 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10dec5f_455a_4587_8c73_6f9c872a0372.slice/crio-2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49 WatchSource:0}: Error finding container 2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49: Status 404 returned error can't find the container with id 2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49 Oct 30 19:19:19.559933 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.559871 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dxxbw" podStartSLOduration=2.150803323 podStartE2EDuration="4.55985266s" totalImagesPullingTime="2.409049337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:19.545723585 +0000 UTC m=+35.998625827" watchObservedRunningTime="2026-10-30 19:19:19.55985266 +0000 UTC m=+36.012754905" Oct 30 19:19:19.560462 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.560433 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz"] Oct 30 19:19:19.861993 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.861740 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:19:19.896329 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.896294 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:19:19.896519 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.896496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:19.901773 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.901754 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:19:19.901904 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.901893 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sz2qj\"" Oct 30 19:19:19.901950 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.901941 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:19:19.901983 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.901967 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:19:19.902037 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902028 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:19:19.902077 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902068 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:19:19.902118 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902109 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:19:19.902211 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902198 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:19:19.902243 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902239 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:19:19.902333 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:19.902323 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:19:20.068785 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.068785 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.068785 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.068996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.069029 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.069027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169683 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169683 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169968 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169968 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169968 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.169968 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.169806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170175 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170175 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170175 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170175 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170362 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170362 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.170362 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.170265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.171292 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.171257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.172009 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.171977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174176 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.173844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174176 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.173910 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174176 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.174000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174176 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.174139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174879 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.174855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.174973 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.174919 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.176347 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.176322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.176462 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.176398 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.176536 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.176515 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.177735 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.177717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.178908 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.178868 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.316237 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.316201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:20.534643 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.534604 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrv7f" event={"ID":"39f57a2e-06f1-4918-a6b3-e93802969d3d","Type":"ContainerStarted","Data":"fd899b290161feca5ada12becc75ba19825968e21852c1b190a6389e34c5452f"} Oct 30 19:19:20.536297 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.536236 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" event={"ID":"2aaa7bfb-b228-4955-9670-bb51ae187fed","Type":"ContainerStarted","Data":"4b594053a82b29ce96112049abb5ca0a280883ea478fd51cc002167a90c41b20"} Oct 30 19:19:20.539238 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.539108 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" event={"ID":"f10dec5f-455a-4587-8c73-6f9c872a0372","Type":"ContainerStarted","Data":"871b0e92f20cbc76bcb1c5c7c2fafc7ffd9069d01f1f5142b0d9a8f5c686f0a6"} Oct 30 19:19:20.539238 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.539154 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" event={"ID":"f10dec5f-455a-4587-8c73-6f9c872a0372","Type":"ContainerStarted","Data":"aa6440a38310d1eeaa4a3999412e372f2e453d358b5f5061f13362af07457a4b"} Oct 30 19:19:20.539238 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.539172 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" event={"ID":"f10dec5f-455a-4587-8c73-6f9c872a0372","Type":"ContainerStarted","Data":"2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49"} Oct 30 19:19:20.542762 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.542732 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fv898" event={"ID":"fa53f8d4-8928-4045-b84f-9d92c83c66e2","Type":"ContainerStarted","Data":"f83bffbcd738e02ac6d7c4ead29a4e3594254271eba52b2c26e1288780170659"} Oct 30 19:19:20.552545 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.552500 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wrv7f" podStartSLOduration=1.876268687 podStartE2EDuration="5.552484669s" totalImagesPullingTime="3.676215982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:20.552335926 +0000 UTC m=+37.005238167" watchObservedRunningTime="2026-10-30 19:19:20.552484669 +0000 UTC m=+37.005386917" Oct 30 19:19:20.626260 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.626204 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fv898" podStartSLOduration=2.233469772 podStartE2EDuration="5.626188176s" totalImagesPullingTime="3.392718404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:20.572596656 +0000 UTC m=+37.025498900" watchObservedRunningTime="2026-10-30 19:19:20.626188176 +0000 UTC m=+37.079090417" Oct 30 19:19:20.626483 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.626465 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:19:20.630511 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:20.630482 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c9150e_8869_4602_b9c3_57ed6659c8f6.slice/crio-5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae WatchSource:0}: Error finding container 5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae: Status 404 returned error can't find the container with id 5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae Oct 30 19:19:20.874393 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.874321 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg"] Oct 30 19:19:20.890818 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.890706 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg"] Oct 30 19:19:20.890966 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.890838 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.896556 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.896476 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:19:20.896556 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.896541 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:19:20.896769 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.896605 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:19:20.896769 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.896671 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:19:20.897620 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.897598 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3cesc8eim54g8\"" Oct 30 19:19:20.897620 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.897599 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-svkv2\"" Oct 30 19:19:20.897760 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.897643 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:19:20.979297 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979429 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979429 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4vg\" (UniqueName: \"kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979552 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979552 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979649 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979649 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979733 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:20.979800 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:20.979776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.090982 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4vg\" (UniqueName: \"kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.091343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.094865 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.092154 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.095748 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.095718 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.102240 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.102033 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.104001 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.103975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4vg\" (UniqueName: \"kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.106246 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.106167 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.106246 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.106206 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.107720 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.107689 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.108349 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.108325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.109201 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.109161 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.323718 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.323682 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:21.547643 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.547608 2575 generic.go:356] "Generic (PLEG): container finished" podID="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" containerID="e0e695210d242f29c42427a054c8c39336c5462dbadbba521da59696fb748707" exitCode=0 Oct 30 19:19:21.548236 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.547695 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5rpc" event={"ID":"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0","Type":"ContainerDied","Data":"e0e695210d242f29c42427a054c8c39336c5462dbadbba521da59696fb748707"} Oct 30 19:19:21.548948 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:21.548925 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae"} Oct 30 19:19:22.205834 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.205461 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg"] Oct 30 19:19:22.289209 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:22.289176 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306fb251_67bf_402f_a8a4_2da67c1b2013.slice/crio-0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206 WatchSource:0}: Error finding container 0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206: Status 404 returned error can't find the container with id 0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206 Oct 30 19:19:22.555355 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.555317 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" event={"ID":"f10dec5f-455a-4587-8c73-6f9c872a0372","Type":"ContainerStarted","Data":"8fec93acb081b1f40e4ea56b04266ce44444fbcd85f19e20e8f13580a63c3adb"} Oct 30 19:19:22.557074 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.557050 2575 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" exitCode=0 Oct 30 19:19:22.557164 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.557144 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} Oct 30 19:19:22.558588 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.558566 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206"} Oct 30 19:19:22.560243 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.560221 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9j28" event={"ID":"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b","Type":"ContainerStarted","Data":"a72cf60f2d69d2e6947c57e6b16ea092a34af7b6ae0239562e1bd4f0c048c0a6"} Oct 30 19:19:22.560337 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.560321 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:22.562408 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.562389 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5rpc" event={"ID":"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0","Type":"ContainerStarted","Data":"093a3c9172761e223977beff4170ba97b28e3bbd348c8245f1f927fef63f7dac"} Oct 30 19:19:22.562496 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.562424 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5rpc" event={"ID":"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0","Type":"ContainerStarted","Data":"e335d1eedaf50155d3417ff9bdc5d98084ec6c44a3cef1b8ec7da2ad6f6f1c79"} Oct 30 19:19:22.564595 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.564576 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" event={"ID":"2aaa7bfb-b228-4955-9670-bb51ae187fed","Type":"ContainerStarted","Data":"a75e7f73801e2ed0e41a05538af9ad79325981dccd2f5b72fe090f4d6ac4315f"} Oct 30 19:19:22.564678 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.564621 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" event={"ID":"2aaa7bfb-b228-4955-9670-bb51ae187fed","Type":"ContainerStarted","Data":"c254659d6acfd3ba21ae7001f067bbca9de7b68bd5afe8e3c22ca8e34782bb03"} Oct 30 19:19:22.564678 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.564655 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" event={"ID":"2aaa7bfb-b228-4955-9670-bb51ae187fed","Type":"ContainerStarted","Data":"4357bb32f0db01b3366b77fe7aae9368d1b6a8b66dcabca1dd5b8d717747c709"} Oct 30 19:19:22.574507 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.574466 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" podStartSLOduration=2.597570543 podStartE2EDuration="4.574435345s" totalImagesPullingTime="1.976864802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.572579274 +0000 UTC m=+39.025481516" watchObservedRunningTime="2026-10-30 19:19:22.574435345 +0000 UTC m=+39.027337588" Oct 30 19:19:22.590889 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.590838 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p5rpc" podStartSLOduration=2.848447377 podStartE2EDuration="4.590820684s" totalImagesPullingTime="1.609192637s" totalInitContainerRuntime="133.18067ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.589916517 +0000 UTC m=+39.042818795" watchObservedRunningTime="2026-10-30 19:19:22.590820684 +0000 UTC m=+39.043722928" Oct 30 19:19:22.606868 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.606814 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h9j28" podStartSLOduration=33.786913463 podStartE2EDuration="38.606797113s" totalImagesPullingTime="4.81988365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.606016525 +0000 UTC m=+39.058918768" watchObservedRunningTime="2026-10-30 19:19:22.606797113 +0000 UTC m=+39.059699356" Oct 30 19:19:22.658105 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:22.658053 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" podStartSLOduration=2.486043747 podStartE2EDuration="4.658040345s" totalImagesPullingTime="2.171996598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:22.657173502 +0000 UTC m=+39.110075745" watchObservedRunningTime="2026-10-30 19:19:22.658040345 +0000 UTC m=+39.110942587" Oct 30 19:19:23.208755 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.208719 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6"] Oct 30 19:19:23.228349 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.228322 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6"] Oct 30 19:19:23.228523 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.228466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.232920 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.232878 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:19:23.233074 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.232962 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:19:23.248090 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.248064 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:19:23.248257 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.248222 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:19:23.248324 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.248290 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2kk4cgcauul0i\"" Oct 30 19:19:23.248382 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.248370 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j6fc8\"" Oct 30 19:19:23.411186 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411371 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7p6t\" (UniqueName: \"kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411371 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411510 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411510 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411510 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.411669 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.411535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513016 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.512975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513016 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7p6t\" (UniqueName: \"kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.513268 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.513177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.514963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.514349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.514963 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.514894 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.515150 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.515030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.516781 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.516757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.517021 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.516966 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.518778 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.518755 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.524179 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.523860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7p6t\" (UniqueName: \"kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.559062 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.559028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:23.578887 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.578860 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-snf6t"] Oct 30 19:19:23.604212 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.604177 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-snf6t"] Oct 30 19:19:23.604329 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.604304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.608215 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.608191 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:19:23.608360 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.608316 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qmmv4\"" Oct 30 19:19:23.616516 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.616475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.616651 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.616614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cwd\" (UniqueName: \"kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.717297 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.717258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.717554 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.717384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cwd\" (UniqueName: \"kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.720685 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.720658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.726356 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.726305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cwd\" (UniqueName: \"kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:23.879426 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.879347 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6"] Oct 30 19:19:23.882982 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:23.882956 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d3aaac_34ae_41ca_883b_f4b6e0bdd642.slice/crio-d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388 WatchSource:0}: Error finding container d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388: Status 404 returned error can't find the container with id d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388 Oct 30 19:19:23.955132 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:23.955104 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:24.256978 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:24.256755 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-snf6t"] Oct 30 19:19:24.264035 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:24.263996 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b270d8_4b48_48c9_a930_99d9c35419cc.slice/crio-cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d WatchSource:0}: Error finding container cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d: Status 404 returned error can't find the container with id cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d Oct 30 19:19:24.576018 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:24.575927 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" event={"ID":"92b270d8-4b48-48c9-a930-99d9c35419cc","Type":"ContainerStarted","Data":"cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d"} Oct 30 19:19:24.577541 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:24.577511 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" event={"ID":"33d3aaac-34ae-41ca-883b-f4b6e0bdd642","Type":"ContainerStarted","Data":"d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388"} Oct 30 19:19:25.027715 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.027685 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:19:25.050997 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.050969 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:19:25.051137 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.051118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.057513 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.057486 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:19:25.061304 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.061285 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:19:25.061421 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.061406 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:19:25.061510 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.061498 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:19:25.070721 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.070698 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:19:25.070832 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.070816 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:19:25.132290 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132290 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132290 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132290 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132471 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132497 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132628 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.132958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.132911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.158341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158290 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:19:25.158341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158295 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8dg65acfiq0rj\"" Oct 30 19:19:25.158341 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158292 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158315 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158486 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158489 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158537 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158541 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158581 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:19:25.158627 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.158602 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gcr25\"" Oct 30 19:19:25.234008 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.233986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234116 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234116 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234497 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234497 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234497 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234497 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234497 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.234854 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.235243 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.235243 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.234879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.235243 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.235110 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.238743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.237978 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.238743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.238251 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.238743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.238615 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.238743 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.238635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239069 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.238860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239124 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.239077 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239504 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.239340 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239593 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.239524 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239857 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.239832 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.239958 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.239838 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.241165 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.240736 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.241688 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.241668 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.241794 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.241774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.242372 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.242351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.242622 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.242601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.242933 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.242885 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.244274 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.244254 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.247057 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.247007 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.248779 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.248666 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.494552 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.494527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:25.594638 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.594406 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} Oct 30 19:19:25.594638 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.594512 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} Oct 30 19:19:25.594638 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.594543 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} Oct 30 19:19:25.594638 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.594568 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} Oct 30 19:19:25.594638 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.594591 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} Oct 30 19:19:25.598669 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.598154 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"befc41a2270b36523af140c7fa15fcd533e941fdfb5789ad3f0bdce9b5158513"} Oct 30 19:19:25.598669 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.598213 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"160cb679af54f07b0bb622acb823c19ba39fbfe78ed7c3285576daf5433c20a3"} Oct 30 19:19:25.598669 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.598238 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"5ab0928e44a74556bf3361921502724a13754ddfce6845e61e42ea49baf38631"} Oct 30 19:19:25.825215 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:25.824909 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:19:25.828705 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:25.828670 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625638da_1fc7_46f4_86f6_f87b66e4cc1f.slice/crio-2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba WatchSource:0}: Error finding container 2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba: Status 404 returned error can't find the container with id 2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba Oct 30 19:19:26.604499 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:26.604391 2575 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" exitCode=0 Oct 30 19:19:26.604872 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:26.604501 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} Oct 30 19:19:26.604872 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:26.604606 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba"} Oct 30 19:19:26.607034 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:26.607006 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" event={"ID":"33d3aaac-34ae-41ca-883b-f4b6e0bdd642","Type":"ContainerStarted","Data":"010c2b37e394243d9f6d7bbbd85e3bb9939b950482995fbb85c9a88d800bd0be"} Oct 30 19:19:26.650209 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:26.650146 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" podStartSLOduration=1.305944116 podStartE2EDuration="3.650128205s" totalImagesPullingTime="2.344184089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:26.649214008 +0000 UTC m=+43.102116262" watchObservedRunningTime="2026-10-30 19:19:26.650128205 +0000 UTC m=+43.103030446" Oct 30 19:19:27.615760 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.615716 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerStarted","Data":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} Oct 30 19:19:27.619393 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.619361 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"f1ca019a8d5a3b0925d2f81788ae12dc5950d56c16361a909f3598fbcb13e2de"} Oct 30 19:19:27.619548 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.619433 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"b83081c1d9fb6837a4f34d3d1fb3a3a8595e3a00bfc9c51ca6144b79bc25908d"} Oct 30 19:19:27.619548 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.619488 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" event={"ID":"306fb251-67bf-402f-a8a4-2da67c1b2013","Type":"ContainerStarted","Data":"16ff14a5505fe7df802722fc588520d0048051d8d9e0135927b73e7146d32faf"} Oct 30 19:19:27.619548 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.619526 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:27.621226 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.621197 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" event={"ID":"92b270d8-4b48-48c9-a930-99d9c35419cc","Type":"ContainerStarted","Data":"5d2407e4d2ae66dd9bbdab313e5c93547a0bd2efbb9f48be496b35e7ed5ffc92"} Oct 30 19:19:27.621367 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.621352 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:27.627201 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.627165 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:27.649422 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.649364 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.32590609 podStartE2EDuration="8.649344917s" totalImagesPullingTime="5.310093985s" totalInitContainerRuntime="13.344842ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:27.648708305 +0000 UTC m=+44.101610547" watchObservedRunningTime="2026-10-30 19:19:27.649344917 +0000 UTC m=+44.102247163" Oct 30 19:19:27.673068 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:27.673015 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" podStartSLOduration=3.55099354 podStartE2EDuration="7.673001126s" totalImagesPullingTime="4.122007586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:27.671340027 +0000 UTC m=+44.124242270" watchObservedRunningTime="2026-10-30 19:19:27.673001126 +0000 UTC m=+44.125903368" Oct 30 19:19:29.544952 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:29.544921 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:29.568662 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:29.568601 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" podStartSLOduration=4.033364572 podStartE2EDuration="6.56858752s" totalImagesPullingTime="2.535222948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:27.695801652 +0000 UTC m=+44.148703895" watchObservedRunningTime="2026-10-30 19:19:29.56858752 +0000 UTC m=+46.021489761" Oct 30 19:19:30.637369 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637331 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} Oct 30 19:19:30.637855 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637400 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} Oct 30 19:19:30.637855 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637428 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} Oct 30 19:19:30.637855 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637468 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} Oct 30 19:19:30.637855 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} Oct 30 19:19:30.637855 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.637522 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerStarted","Data":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} Oct 30 19:19:30.687212 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:30.687154 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.648832697 podStartE2EDuration="6.687135329s" totalImagesPullingTime="3.025379143s" totalInitContainerRuntime="12.923489ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:30.684498675 +0000 UTC m=+47.137400916" watchObservedRunningTime="2026-10-30 19:19:30.687135329 +0000 UTC m=+47.140037571" Oct 30 19:19:33.632386 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:33.632356 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:35.495940 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:35.495891 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:41.306321 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:41.306292 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:43.559529 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:43.559492 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:43.559944 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:43.559570 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:45.822745 ip-10-0-132-153 kubenswrapper[2575]: E1030 19:19:45.822708 2575 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:19:48.762382 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.762327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:48.764863 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.764846 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:19:48.775001 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.774968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:48.863263 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.863215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:48.865897 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.865880 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:19:48.876083 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.876052 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:48.946020 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.945984 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-t757m\"" Oct 30 19:19:48.953645 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.953624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:48.971718 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.971686 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xs9gl\"" Oct 30 19:19:48.979667 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:48.979637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:49.231224 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:49.231085 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n84k8"] Oct 30 19:19:49.233866 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:49.233835 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4738657b_e0c8_4a35_9893_85f99f3632b2.slice/crio-b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487 WatchSource:0}: Error finding container b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487: Status 404 returned error can't find the container with id b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487 Oct 30 19:19:49.263663 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:49.263634 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dmtjr"] Oct 30 19:19:49.266649 ip-10-0-132-153 kubenswrapper[2575]: W1030 19:19:49.266626 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b81ec2a_4636_47b1_93dd_c38322d944f7.slice/crio-ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d WatchSource:0}: Error finding container ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d: Status 404 returned error can't find the container with id ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d Oct 30 19:19:49.716114 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:49.716077 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmtjr" event={"ID":"3b81ec2a-4636-47b1-93dd-c38322d944f7","Type":"ContainerStarted","Data":"ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d"} Oct 30 19:19:49.717210 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:49.717168 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n84k8" event={"ID":"4738657b-e0c8-4a35-9893-85f99f3632b2","Type":"ContainerStarted","Data":"b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487"} Oct 30 19:19:50.723421 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:50.723375 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmtjr" event={"ID":"3b81ec2a-4636-47b1-93dd-c38322d944f7","Type":"ContainerStarted","Data":"be7e47c7ca8b03cafb146d8b38bd281c9d4de30c3c5945b95fc543661bc9007d"} Oct 30 19:19:50.723846 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:50.723511 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dmtjr" event={"ID":"3b81ec2a-4636-47b1-93dd-c38322d944f7","Type":"ContainerStarted","Data":"97ba182879ca82ef7cb1d8dffbade5c0acfe86b40de4a311a3c95931146a31a0"} Oct 30 19:19:50.741195 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:50.741135 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dmtjr" podStartSLOduration=65.575604822 podStartE2EDuration="1m6.74111592s" totalImagesPullingTime="1.165511098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:50.739522059 +0000 UTC m=+67.192424311" watchObservedRunningTime="2026-10-30 19:19:50.74111592 +0000 UTC m=+67.194018163" Oct 30 19:19:53.573111 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:53.573076 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:53.961111 ip-10-0-132-153 kubenswrapper[2575]: I1030 19:19:53.961004 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:19:53.961174 ip-10-0-132-153 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:19:53.975814 ip-10-0-132-153 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:19:53.975991 ip-10-0-132-153 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:19:53.976173 ip-10-0-132-153 systemd[1]: kubelet.service: Consumed 4.752s CPU time. Oct 30 19:19:53.991023 ip-10-0-132-153 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:19:54.052853 ip-10-0-132-153 kubenswrapper[6907]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:54.052853 ip-10-0-132-153 kubenswrapper[6907]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:19:54.052853 ip-10-0-132-153 kubenswrapper[6907]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:54.052853 ip-10-0-132-153 kubenswrapper[6907]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:19:54.055243 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055134 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.055243 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055237 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.055243 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055242 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.055243 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055247 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055255 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055259 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055263 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055267 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055271 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055275 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055279 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055282 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055286 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055290 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055295 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055299 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055302 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055306 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055310 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055314 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055318 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055322 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055326 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.055580 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055330 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055335 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055340 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055344 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055348 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055352 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055356 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055360 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055364 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055368 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055372 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055376 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055380 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055385 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055389 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055393 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055398 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055402 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055407 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055411 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.056476 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055416 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055420 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055424 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055431 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055438 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055462 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055466 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055470 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055474 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055478 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055482 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055486 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055491 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055497 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055501 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055506 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055510 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055515 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055519 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055523 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.057420 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055527 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055532 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055536 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055541 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055545 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055550 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055555 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055559 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055565 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055570 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055574 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055578 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055582 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055586 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055590 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055595 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055599 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055605 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055610 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.058299 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055614 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055619 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055623 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055628 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055632 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055638 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055643 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055648 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055652 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055657 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055661 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055666 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055670 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055675 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055682 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055688 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055693 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055698 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055704 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.059163 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055709 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055713 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055717 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055722 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.055726 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057026 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057039 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057044 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057049 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057054 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057059 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057063 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057068 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057073 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057077 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057081 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057086 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057096 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057101 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057106 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.060041 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057110 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057115 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057119 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057124 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057128 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057132 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057137 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057141 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057145 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057150 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057154 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057159 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057163 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057167 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057171 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057176 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057180 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057184 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057188 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.060989 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057193 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057199 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057203 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057207 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057211 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057215 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057220 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057224 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057228 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057232 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057239 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057244 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057249 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057256 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057261 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057265 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057270 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057274 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057280 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057284 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.061784 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057289 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057293 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057298 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057302 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057309 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057316 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057322 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057327 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057331 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057335 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057340 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057344 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057348 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057353 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057357 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057361 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057366 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057370 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057374 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.062341 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057379 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057384 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057388 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057394 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057400 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057404 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057408 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057413 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057417 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057421 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057426 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057430 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057434 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057438 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057461 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057465 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057469 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057474 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057478 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057483 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.062852 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057487 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057491 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057495 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057500 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057504 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057508 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057512 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057517 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057523 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057527 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057532 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057536 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.057540 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058413 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058426 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058433 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058455 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058460 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058465 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.063360 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058469 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058474 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058478 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058482 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058486 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058491 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058495 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058499 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058504 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058508 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058513 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058517 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058521 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058525 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058530 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058534 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058538 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058542 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058547 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058551 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.064089 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058555 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058560 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058564 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058568 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058572 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058576 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058581 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058586 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058591 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058595 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058600 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058604 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058608 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058612 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058617 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058621 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058627 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058631 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058635 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058640 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.064934 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058644 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058648 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058652 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058656 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058661 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058665 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058669 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058673 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058677 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058683 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058687 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058691 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058696 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058700 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058705 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058709 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058713 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058717 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058723 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.065693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058732 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058737 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058742 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058746 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058751 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058755 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058759 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058763 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058768 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058773 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058777 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058781 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058785 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058790 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058794 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058798 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058803 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058809 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058816 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.066185 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058821 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058825 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058833 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058838 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058843 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058847 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058851 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058856 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058861 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058866 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058870 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058874 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058878 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058883 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058895 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058900 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058905 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058909 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058913 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058917 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.066674 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058922 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.058926 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059066 6907 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059078 6907 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059087 6907 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059095 6907 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059102 6907 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059107 6907 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059114 6907 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059120 6907 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059125 6907 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059131 6907 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059136 6907 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059141 6907 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059149 6907 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059155 6907 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059159 6907 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059164 6907 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059169 6907 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059173 6907 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059182 6907 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059187 6907 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059192 6907 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059197 6907 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:19:54.067201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059203 6907 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059213 6907 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059218 6907 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059223 6907 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059229 6907 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059234 6907 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059239 6907 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059244 6907 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059249 6907 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059253 6907 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059260 6907 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059265 6907 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059270 6907 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059275 6907 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059280 6907 flags.go:64] FLAG: --enable-server="true" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059285 6907 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059292 6907 flags.go:64] FLAG: --event-burst="100" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059297 6907 flags.go:64] FLAG: --event-qps="50" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059301 6907 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059307 6907 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059312 6907 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059319 6907 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059326 6907 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059331 6907 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059336 6907 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:19:54.067799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059341 6907 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059346 6907 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059351 6907 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059356 6907 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059360 6907 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059365 6907 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059370 6907 flags.go:64] FLAG: --feature-gates="" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059380 6907 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059390 6907 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059395 6907 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059400 6907 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059406 6907 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059411 6907 flags.go:64] FLAG: --help="false" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059416 6907 flags.go:64] FLAG: --hostname-override="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059422 6907 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059427 6907 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059432 6907 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059438 6907 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059463 6907 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059468 6907 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059473 6907 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059477 6907 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059482 6907 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:19:54.068432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059487 6907 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059493 6907 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059498 6907 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059503 6907 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059507 6907 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059512 6907 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059519 6907 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059525 6907 flags.go:64] FLAG: --lock-file="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059530 6907 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059535 6907 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059540 6907 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059549 6907 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059555 6907 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059559 6907 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059565 6907 flags.go:64] FLAG: --logging-format="text" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059570 6907 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059576 6907 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059582 6907 flags.go:64] FLAG: --manifest-url="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059587 6907 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059594 6907 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059599 6907 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059606 6907 flags.go:64] FLAG: --max-pods="110" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059612 6907 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059617 6907 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059621 6907 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:19:54.069092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059627 6907 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059632 6907 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059637 6907 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059642 6907 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059653 6907 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059658 6907 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059663 6907 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059668 6907 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059673 6907 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059677 6907 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059682 6907 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059687 6907 flags.go:64] FLAG: --port="10250" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059692 6907 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059700 6907 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-03d09ec56d3790992" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059705 6907 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059710 6907 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059715 6907 flags.go:64] FLAG: --register-node="true" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059720 6907 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059726 6907 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059731 6907 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059735 6907 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059740 6907 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059748 6907 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059753 6907 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059760 6907 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059765 6907 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:19:54.069721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059770 6907 flags.go:64] FLAG: --runonce="false" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059775 6907 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059780 6907 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059786 6907 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059791 6907 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059796 6907 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059802 6907 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059807 6907 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059812 6907 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059816 6907 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059821 6907 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059826 6907 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059830 6907 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059836 6907 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059841 6907 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059845 6907 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059854 6907 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059858 6907 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059863 6907 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059873 6907 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059878 6907 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059882 6907 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059887 6907 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059892 6907 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059897 6907 flags.go:64] FLAG: --v="2" Oct 30 19:19:54.070496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059904 6907 flags.go:64] FLAG: --version="false" Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059910 6907 flags.go:64] FLAG: --vmodule="" Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059916 6907 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.059922 6907 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060160 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060169 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060174 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060178 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060183 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060187 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060191 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060196 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060201 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060206 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060210 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060214 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060219 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060223 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060227 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060232 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.071148 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060236 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060241 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060245 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060249 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060253 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060257 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060265 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060270 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060274 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060279 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060283 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060287 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060292 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060296 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060300 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060305 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060310 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060315 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060320 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060324 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.071659 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060328 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060332 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060337 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060341 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060346 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060351 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060355 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060362 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060367 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060372 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060377 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060382 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060387 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060391 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060396 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060401 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060406 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060410 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060416 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060421 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.072196 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060425 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060430 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060434 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060454 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060459 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060464 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060468 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060473 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060477 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060481 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060485 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060489 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060494 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060499 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060503 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060507 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060511 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060517 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060521 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.072855 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060525 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060530 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060534 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060540 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060546 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060551 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060556 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060560 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060564 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060569 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060573 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060580 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060585 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060589 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060593 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060598 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060602 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060606 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060611 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.073321 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060615 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060619 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060624 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060628 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060633 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060637 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060641 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060646 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060651 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060656 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060660 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060665 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.060708 6907 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060766 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060774 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.073868 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060780 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060786 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060790 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060795 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060799 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060803 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060808 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060815 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060819 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060823 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060829 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060836 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060842 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060847 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060852 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060856 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060861 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060865 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060869 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060874 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.074267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060878 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060883 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060887 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060892 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060896 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060900 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060905 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060910 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060914 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060918 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060924 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060928 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060933 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060937 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060941 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060945 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060950 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060954 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060959 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.074829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060965 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060969 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060974 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060979 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060984 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060988 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060992 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.060998 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061005 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061009 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061014 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061018 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061023 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061028 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061032 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061036 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061041 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061045 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061050 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.075342 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061054 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061058 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061062 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061067 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061071 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061076 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061081 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061086 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061090 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061094 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061098 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061105 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061110 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061116 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061121 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061125 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061132 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061136 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061141 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.075879 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061145 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061149 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061154 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061158 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061163 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061167 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061172 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061176 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061181 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061185 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061189 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061194 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061198 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061202 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061207 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061211 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061215 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061220 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061224 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061229 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.076379 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061233 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061239 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061243 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061248 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061253 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061257 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.061263 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.067042 6907 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.067059 6907 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.067078 6907 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.067083 6907 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067146 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067151 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067156 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067159 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067162 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067165 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067168 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067170 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.077019 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067173 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067176 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067178 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067182 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067185 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067187 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067190 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067192 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067196 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067199 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067201 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067204 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067207 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067210 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067213 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067216 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067219 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067222 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067224 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.077581 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067232 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067235 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067238 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067241 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067244 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067246 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067249 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067251 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067254 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067257 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067260 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067263 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067265 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067268 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067270 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067273 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067276 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067278 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067281 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067284 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.078046 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067287 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067291 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067295 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067299 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067302 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067305 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067307 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067310 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067313 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067317 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067320 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067322 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067325 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067328 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067331 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067333 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067336 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067338 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067341 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.078536 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067344 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067346 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067349 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067352 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067355 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067357 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067360 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067363 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067367 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067371 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067374 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067377 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067379 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067383 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067386 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067389 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067392 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067394 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067397 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.079283 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067399 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067402 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067405 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067408 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067410 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067413 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067416 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067419 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067421 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067424 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067426 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067429 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067431 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067434 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067437 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067439 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067457 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067461 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067465 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067467 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.080011 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067470 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.067500 6907 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067649 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067656 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067659 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067661 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067666 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067671 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067674 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067677 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067680 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067683 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067685 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067688 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067713 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.080813 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067717 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067720 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067722 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067725 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067728 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067730 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067733 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067736 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067739 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067741 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067744 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067747 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067749 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067752 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067754 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067757 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067759 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067762 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067764 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.081247 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067767 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067770 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067773 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067775 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067778 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067780 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067784 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067787 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067790 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067792 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067795 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067798 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067801 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067804 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067809 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067812 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067815 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067818 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067820 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.081843 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067823 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067826 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067828 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067831 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067833 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067836 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067838 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067841 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067844 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067846 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067849 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067851 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067854 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067856 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067858 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067861 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067864 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067867 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067869 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067872 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.082486 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067875 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067878 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067880 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067883 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067888 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067891 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067893 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067896 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067899 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067901 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067904 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067906 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067909 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067911 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067914 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067917 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067919 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067922 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067924 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067927 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.083152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067929 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067937 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067941 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067943 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067946 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067949 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067951 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067954 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067957 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067959 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067962 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067965 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067968 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067981 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.067984 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.068015 6907 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:19:54.083825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068052 6907 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068057 6907 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068060 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068063 6907 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068066 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068069 6907 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068072 6907 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068075 6907 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068077 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068080 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068083 6907 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068085 6907 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068088 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068091 6907 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068093 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068096 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068099 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068101 6907 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068104 6907 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:19:54.084330 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068106 6907 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068109 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068111 6907 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068114 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068116 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068119 6907 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068122 6907 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068124 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068127 6907 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068130 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068133 6907 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068136 6907 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068139 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068142 6907 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068145 6907 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068148 6907 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068150 6907 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068153 6907 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068156 6907 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:19:54.084829 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068159 6907 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068161 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068165 6907 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068169 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068173 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068175 6907 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068178 6907 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068181 6907 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068184 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068187 6907 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068190 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068192 6907 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068195 6907 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068197 6907 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068200 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068202 6907 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068205 6907 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068207 6907 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068210 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:19:54.085349 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068214 6907 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068217 6907 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068219 6907 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068222 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068225 6907 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068228 6907 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068232 6907 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068238 6907 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068242 6907 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068246 6907 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068261 6907 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068264 6907 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068267 6907 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068269 6907 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068272 6907 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068274 6907 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068277 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068280 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068282 6907 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068285 6907 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:19:54.085990 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068287 6907 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068290 6907 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068293 6907 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068295 6907 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068298 6907 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068301 6907 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068303 6907 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068306 6907 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068308 6907 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068311 6907 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068314 6907 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068316 6907 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068320 6907 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068323 6907 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068326 6907 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068328 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068331 6907 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068333 6907 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068336 6907 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068341 6907 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:19:54.086519 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068344 6907 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068347 6907 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068350 6907 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068352 6907 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068355 6907 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068358 6907 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068360 6907 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068363 6907 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:19:54.068368 6907 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.068424 6907 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069471 6907 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069529 6907 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069720 6907 server.go:1014] "Starting client certificate rotation" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069868 6907 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069931 6907 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:13:43 +0000 UTC" deadline="2028-07-04 12:22:48.427544111 +0000 UTC" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.069971 6907 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14705h2m54.357577005s" Oct 30 19:19:54.087010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.070250 6907 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:19:54.087435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.071382 6907 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:19:54.087435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.073029 6907 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:19:54.087435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.074991 6907 log.go:25] "Validated CRI v1 image API" Oct 30 19:19:54.087435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.075690 6907 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:19:54.087435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.078202 6907 fs.go:135] Filesystem UUIDs: map[0cf2dfd4-42d6-4be2-ab48-4f729b1e9faf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe513efd-079e-424e-a28b-6958259b3539:/dev/nvme0n1p3] Oct 30 19:19:54.087621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.078218 6907 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba/userdata/shm major:0 minor:419 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b594053a82b29ce96112049abb5ca0a280883ea478fd51cc002167a90c41b20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b594053a82b29ce96112049abb5ca0a280883ea478fd51cc002167a90c41b20/userdata/shm major:0 minor:255 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da/userdata/shm major:0 minor:56 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487/userdata/shm major:0 minor:486 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d88c390d5522878eb56454aba7836af7b80cf176c8de256480b890590df8bc8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d88c390d5522878eb56454aba7836af7b80cf176c8de256480b890590df8bc8b/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d/userdata/shm major:0 minor:488 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01332dde-ec0a-41fd-abfc-f73fab46b857/volumes/kubernetes.io~projected/kube-api-access-zz44v:{mountpoint:/var/lib/kubelet/pods/01332dde-ec0a-41fd-abfc-f73fab46b857/volumes/kubernetes.io~projected/kube-api-access-zz44v major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/alertmanager/9 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~empty-dir/config-out major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/kube-api-access-dfbrh:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/kube-api-access-dfbrh major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/tls-assets major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/config-volume major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/web-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~projected/kube-api-access-j5scm:{mountpoint:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~projected/kube-api-access-j5scm major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~projected/kube-api-access-4mhlq:{mountpoint:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~projected/kube-api-access-4mhlq major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~projected/kube-api-access-9b4vg:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~projected/kube-api-access-9b4vg major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~projected/kube-api-access-b7p6t:{mountpoint:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~projected/kube-api-access-b7p6t major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~projected/kube-api-access-mc8h8:{mountpoint:/var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~projected/kube-api-access-mc8h8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~projected/kube-api-access-j5cw9:{mountpoint:/var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~projected/kube-api-access-j5cw9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~secret/metrics-certs major:0 minor:485 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4738657b-e0c8-4a35-9893-85f99f3632b2/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4738657b-e0c8-4a35-9893-85f99f3632b2/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:481 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~projected/kube-api-access-84bxs:{mountpoint:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~projected/kube-api-access-84bxs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~projected/kube-api-access-7qs26:{mountpoint:/var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~projected/kube-api-access-7qs26 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/config-reloader/2 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/prometheus/15 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/kube-api-access-5qb88:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/kube-api-access-5qb88 major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/tls-assets major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/config major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/web-config major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8dcea2b7-e349-4e35-b71a-35ad85f61c54/volumes/kubernetes.io~projected/kube-api-access-rtfv2:{mountpoint:/var/lib/kubelet/pods/8dcea2b7-e349-4e35-b71a-35ad85f61c54/volumes/kubernetes.io~projected/kube-api-access-rtfv2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~projected/kube-api-access-q9cwd:{mountpoint:/var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~projected/kube-api-access-q9cwd major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d167a43-62ce-4e3b-98ad-a095c3276ea8/volumes/kubernetes.io~projected/kube-api-access-fqfcd:{mountpoint:/var/lib/kubelet/pods/9d167a43-62ce-4e3b-98ad-a095c3276ea8/volumes/kubernetes.io~projected/kube-api-access-fqfcd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/volumes/kubernetes.io~projected/kube-api-access-4srnf:{mountpoint:/var/lib/kubelet/pods/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/volumes/kubernetes.io~projected/kube-api-access-4srnf major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c96d731e-41cf-442f-adfc-3d36a9908b31/volumes/kubernetes.io~projected/kube-api-access-2jrpc:{mountpoint:/var/lib/kubelet/pods/c96d731e-41cf-442f-adfc-3d36a9908b31/volumes/kubernetes.io~projected/kube-api-access-2jrpc major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb/volumes/kubernetes.io~projected/kube-api-access-kkh7w:{mountpoint:/var/lib/kubelet/pods/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb/volumes/kubernetes.io~projected/kube-api-access-kkh7w major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b/volumes/kubernetes.io~projected/kube-api-access-vjz6k:{mountpoint:/var/lib/kubelet/pods/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b/volumes/kubernetes.io~projected/kube-api-access-vjz6k major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~projected/kube-api-access-wglxr:{mountpoint:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~projected/kube-api-access-wglxr major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~projected/kube-api-access-xc7w2:{mountpoint:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~projected/kube-api-access-xc7w2 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~projected/kube-api-access-9nzwt:{mountpoint:/var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~projected/kube-api-access-9nzwt major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/0b8319c4eb174a0a30b3f12035e2c2445c004bfee0ed7d242ea7f01f7db9697a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5d9adc965aa7de898e60b7acf8637564cdead0fc9e828cc5e35135b55f303823/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/09c8a187f76bb0b052b22a5b1bd83909d51fda12f2a33c2e2da5a70b7ce3eec7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/ce52c365c815c866883ceb6b5e543da9998bc2502432368a764d27563943e032/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/b2e24c82aa1197d652b9236b9494eac3b765b44ba227595b3433d53aa2a40a3c/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/07fcd0a114846dabab4967c1f978728248eab3c14a9aa46c6db5ea0515c49aa5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/79aaf1ba440bffa38a3d069b99138e81adc1078db6c56866896942e7bef6a0c6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/f38aab3f1a3b2c9b7918ec3fe2c391b88c743a7ff0747befc5d1fd1359403ce2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5808da58bd477f3138626ee42506048f2d1a091d30325f6db47b01dd5f5b980d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3a023c2e2e9112e276bf5b024849499421211d05ae98391d3c6feeccb51bab24/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/dff54eb1eaab8bd9c911d11a2aaed90541d83ca59a1ac7d430ce693885a72387/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/c79c54ca07c32f12c6d0a2cf14e25b0f4f783949b102b13cb4549a094846fd15/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/e5a63e880bf93520869989021140a4ceee7099432a3880a8f5855304f34c4a11/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/960081b843b77f437045ed41dafe0582a11ddda6fb3cd37aa20e7f2a87be1244/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/22b753ad8dc708964d662fe5c89ff2698b641637adda4ebc23d074e15aba9b01/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4e3cd54a7b5974899b275f188af0ef61158ca55cdda4165b97dbc873d6f1f133/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d9bc4f9beb703a2daa80ab9db039f0e21c97958eb936e6790844c0dcfd8a7d68/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3df87dab76970bdf825cdba8f86d42d20d4112a77a362fb8f02823a773efecbd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a48ad1935fb46d57209e10d14b8fea37d4e5a5355c269e5c49d0447071fc7f73/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/96e82278088a10257059acd0b2daf80b7c652c70f4638787c96c96633c44c7be/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/40010952e8665898670b65dea59f467d361a7b3a851595ccfd09114d3992cd5c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3e1f39644734f009a04fce96ac463ba1161b0e8bb6d26645708d8d40c0aa4e94/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2bf00df15b75d01c0d8218de4a2c6b2e537d6167cb26d953e56bc42779aba767/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0b1d183f9bc1cb2e447b47f3ce81dd31bed2cfc1ef10f1410597c9f34ea150b2/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/b6c8577bd1a7f1038e59ffc8d213f83ef3dbab0d6cc31a9698ed5d1f76a358ee/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/5153fd3898e03f597fd349f3ef1b01e2ff9dd44806f5c2573726d5cfd6a713bf/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/d820504d570bc05556bb81581125bf1a46ee827ab72d620030732af14ff00391/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4c9a71377d72b3fa699a16a14b6f9d96a7157c5e98a1fec0e03cbf5bbb58bbb1/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/843a901db1b14eae6d571efc3b3bf6741c0392a6e17fd68e14519a788a07b32a/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/4c4079ca504119c80a6b4521eeef408b4650978c02d1615e8cb9c8602d142088/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/96d91e6d71367e3d267611aa1cf06fd11e42da6415329e867c9064fcbeb713bf/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f63052a9e9077cca0bb0a27660bdd618539b3056ae8bacf7446148a9d9aa2484/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/914ed539857661c679dc646327313056c229d9e8ce1cd8b92159319ac8bfbe43/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/34bb98e43ab7cdfb13a36c38a1302275bcb8e91a9ef1052f5ee6aa4a59a9d607/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/39fa6bfa7ebaabe199cbf1159d76dc5cc65ace623d3a0bd892769b8eb787e4a4/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/ce5556d26f99db079149563fb1f0f20bca86dddd1fd372d973e71146609c53a1/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/e138c6b82152e7e6213ff87f0d56e398cad6924ec50e10c08ba69b82b0e5deec/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/fdfbebdc508ee0843075d770722af69dd3af58d90afc8d5a0da1dda7503bfa50/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/5d5eec9cd91aae991ab65857e2fbe6f192145b9622dee2427c8ea6e66f0d158c/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/1c1b96c6840ee4e9bb5d74fbfa2e4ec9aa5f9ad59bd42a96f9af681285bfc1d4/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/761a4122dc9580eb1518fd605d6822f573023355847e7ede9a9eaf17ea935fe4/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/6b156d10ecdbde807a23d3df8e0ada9f39bdf5a4149daf3b9158f89213d89fa0/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/0a153f1dd8f5d9c6b405277a3c9f4f5922af60dfe2cfc396f54703fd2e4d95a6/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/78a5875aa9cafb592a1ce439ad0362f3a5af53ea88970a14bdba48c079f14101/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/03e96420fb81e447fa8ed05ce45a7a36965a01898bdacf81729737b056ef9629/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/2f125e272391d3b317f555ad7c8cfc8ce63a250c3de38de3de5a7911fcd61a28/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/0cc4602c3126664b37d316903b6bca94e59a8fcc5a3f107c23f206c2accd8688/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/31742fbb42b1dabb81eac15f923e910362fed93bef9e3c3ab1bf1127e22ddd7c/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/b3321569d2e33688ce54dab97edafb7eadca9c4eed2ea07a313a7c65c487a7d7/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/e72f0899614081c8c876bf5eb34d7906800f08e4d1036e1fb5e6f10950fa8537/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/bb6c00ae84bd551a71269dec985f3c1cb2bce63441e131f47c1c8fc78d15f2d6/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/51a4c3d9865ca64322f6232750f375376f8fd7305054194f274a2dd50e2e7e4b/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/b156b20c30318f58f1f3718799a43b95e2e2adfc4916b47d0bdb3e01e2038f45/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/9b0ea6dea88325292e6ed0d82f9feebdabecf0288be9dbb5e2f9837599af7f5b/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/880826e63b7c9c031b41dfb15e1d088207ff1b39f55ee9654559338a0566ad99/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/14b13bb751e4601b0800adb96cda7dda1094f2f9fc22be874f4c1ab0cf40458e/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/aae9a563e268c837bc1d3be30c67e62a2ad602986951d93cfda4ce1598149852/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/7fdbcd503f6ed5c3bc6cc64f97365e24d3e374aa8a29ac36ebf8822ffab21e8f/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/884a2e24bc6c5a1f6f691ed7c805b7f3e3ec7f7af35f72c3c9055d7928b37855/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/b9a197e98d4b705f02483fe7b2a5599384ce8d4043ab4ebd118e9410b6e61f6a/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/48d7085cc0342b604621dd4a8bf4a699b84eeb5554881ebea5e1b00a7e19dfb1/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/b2d95d26710e82921d110cb2c65633c542dd0f2e25f90a62599913bbf02e596e/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/c79a508b06252f955c3d7583fe4cc068f0ffc6d85b863aa3eda167ecfd88204f/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/b62fc80d721d614f4753a6ba223b7b709f1222aceceb495f21377cf21214c1e9/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/77f1e49e97298a4f7a9a15de68412c246720505ddbd55322a4c570c06ccae1b4/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/37e2c37af20a52ea07f74b89a123f5775c3dc5c19ccd73e8b31aa126d8000e50/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/22d424a6cbcffad8e92a6c2fb14e8a6447161fc8086dada140d647fea8dc0e30/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/bb8b2d4825c938bd8d6f832a2136faa84d8c2c7ed8c3107499f6861e572d3f10/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/e21373b9a70265495c2bd36e928629fd80237c608c3319a0e772dab2b12d0732/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-492:{mountpoint:/var/lib/containers/storage/overlay/b0eae30ce50203b062812150c9a1d109df4d3d05c17af2d9eecc7e12ad967448/merged major:0 minor:492 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/7b802ed3fc00018b3dddfbb08a191cdcd24e0717c732ebf56524b20f258d6b24/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/094cf225cbdb84472ec6b1e552efba636cc70a84e8b6ae7241c6913439e35981/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/a8fb584f8847687f551cee6fab741f6d93872a1242f4953c9f2f36b3761efa64/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-58:{mountpoint:/var/lib/containers/storage/overlay/9fa44234343907d8a7286831cfc019712453c2dc34a03522bb82e3e3073587da/merged major:0 minor:58 fsType:overlay blockSize:0} overlay_0-65:{mountpoint:/var/lib/containers/storage/overlay/33d0fa1c74157e771fb285ce987064337bd57c56d9730057049672551604b936/merged major:0 minor:65 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0456c1ede5cf304a078d75c19503957adaed738971269d91dfcc250814d23666/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/373a51c3d203495969b32da2ed2e10321b02cec8b75202d2e0cacd258f52015a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/30038d0e89e9c09768f17ae9ad32272d8defed7f94bfdcd4e7000920061c83b2/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/5678be3c27a63e71fc47b0cc4f683d8e1dbcfc396b6e1c105459e31e6ae7023d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d2175bca4cdba06fa3a03f627246c51aec041b8a447c9b520ca4ad79c5efc9d6/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a29b2304e1fdca3fcdbe512d1140ff0253ee9dd2ed9769da5cab8a8af107035e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7be11e591c0d8d6c61de40e651fb0227cc520f51ad264c90eb30a949e406d2cb/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f7282809b41b758a5a13e5954de6cb17386e424f47b401ba51f59c01ef95b5d3/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:19:54.089093 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.088647 6907 manager.go:217] Machine: {Timestamp:2026-10-30 19:19:54.088134427 +0000 UTC m=+0.064360361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101027 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fb9645611140446ba102e536df658 SystemUUID:ec2fb964-5611-1404-46ba-102e536df658 BootID:60900f46-05e4-474d-824d-d24ba23b8e7e Filesystems:[{Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfbe8234d4bbc9369a1b5d2974913fc90ce3b1324d3ca8d3ac330b79d9ab2d2d/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01332dde-ec0a-41fd-abfc-f73fab46b857/volumes/kubernetes.io~projected/kube-api-access-zz44v DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-58 DeviceMajor:0 DeviceMinor:58 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b594053a82b29ce96112049abb5ca0a280883ea478fd51cc002167a90c41b20/userdata/shm DeviceMajor:0 DeviceMinor:255 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~projected/kube-api-access-xc7w2 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/860aa432073c29c860f92ed21f8bf00b7a7ba38937d3d00d7c20dbe828db73ce/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~projected/kube-api-access-wglxr DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~projected/kube-api-access-4mhlq DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:380 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~projected/kube-api-access-mc8h8 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4738657b-e0c8-4a35-9893-85f99f3632b2/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:481 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d708773746750d0f3446a4396a3bef0088433d9fdc7f0be78ede248c7bb6f388/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d167a43-62ce-4e3b-98ad-a095c3276ea8/volumes/kubernetes.io~projected/kube-api-access-fqfcd DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0523232a56b276f2e9effb5a11f86c1bfb5ebf00967079aadcefadc7a9227206/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2113598b09187be2cb09792610753520c2926ad1cfedfcfe261338fb5f595a49/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d88c390d5522878eb56454aba7836af7b80cf176c8de256480b890590df8bc8b/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba/userdata/shm DeviceMajor:0 DeviceMinor:419 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:379 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5364bcd9-a94f-4072-973c-bc3d9993d2bd/volumes/kubernetes.io~projected/kube-api-access-7qs26 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff5e186750cb7c49f5e324711f32f35fae3a1314b0f7d978a3780c231645794e/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1557c5fd0a38788a997f67483854524c02b208f871cf452a57a9d92a16a9195/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/573c4176d345fe18cd44fa82149977ebe85b39f97dda8f722779b3b4830b9c62/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92b270d8-4b48-48c9-a930-99d9c35419cc/volumes/kubernetes.io~projected/kube-api-access-q9cwd DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:390 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e5dcce8-8409-491b-8428-eecdaf1ea2c0/volumes/kubernetes.io~projected/kube-api-access-84bxs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae8882d35f393489e27b0cb4cc25dc1e69ec6c9136cfaf5aca7a26a4ac746c8f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~projected/kube-api-access-dfbrh DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:485 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bd9779bc021cfb69166c3a8451b747da54d6dbda2abd82a71053f1fc02067b8/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed105230a3559775c56c8c62d664792e83a22bfec537e4557531b86c1510ca9d/userdata/shm DeviceMajor:0 DeviceMinor:488 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0e5fb759dd8c526d13cf71daeba2d61fcb5b88ffe72de0cab418d033fa2dea9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da/userdata/shm DeviceMajor:0 DeviceMinor:56 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~projected/kube-api-access-9b4vg DeviceMajor:0 DeviceMinor:311 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:389 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b/volumes/kubernetes.io~projected/kube-api-access-vjz6k DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/39f57a2e-06f1-4918-a6b3-e93802969d3d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2aaa7bfb-b228-4955-9670-bb51ae187fed/volumes/kubernetes.io~projected/kube-api-access-j5scm DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-492 DeviceMajor:0 DeviceMinor:492 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c96d731e-41cf-442f-adfc-3d36a9908b31/volumes/kubernetes.io~projected/kube-api-access-2jrpc DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:292 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~projected/kube-api-access-b7p6t DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~projected/kube-api-access-5qb88 DeviceMajor:0 DeviceMinor:387 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2884df347214fc87fcf0fa9075345aaf1a69db32365ddcce39b192ec6b30487/userdata/shm DeviceMajor:0 DeviceMinor:486 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb/volumes/kubernetes.io~projected/kube-api-access-kkh7w DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-65 DeviceMajor:0 DeviceMinor:65 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:307 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:346 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33d3aaac-34ae-41ca-883b-f4b6e0bdd642/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:386 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8dcea2b7-e349-4e35-b71a-35ad85f61c54/volumes/kubernetes.io~projected/kube-api-access-rtfv2 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3b81ec2a-4636-47b1-93dd-c38322d944f7/volumes/kubernetes.io~projected/kube-api-access-j5cw9 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c8e7c82fc9aa71fab2dd8282db4784c245517247ca5ff2d8ff1038c20e6faf0/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f9ad062ffe69290d5eb0a6309e93e1e9b02bf2f9e97f2ffda5d5aca02e69f34/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca26215128e1152c854e0a0a3412b0dad77aefb54ecd429584ae71e7c8645636/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~projected/kube-api-access-9nzwt DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:384 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b86255da2cb1c6253bf18977cd353f9ef980c9fe4eda24fd880f682861523dc9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fa53f8d4-8928-4045-b84f-9d92c83c66e2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:308 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/volumes/kubernetes.io~projected/kube-api-access-4srnf DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f10dec5f-455a-4587-8c73-6f9c872a0372/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/08952057fbef30998a62182cbd871064c89850cddf8dcec21dedaca170ecb1b5/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/306fb251-67bf-402f-a8a4-2da67c1b2013/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0523232a56b276f MacAddress:ce:aa:14:39:4f:bc Speed:10000 Mtu:8901} {Name:2113598b09187be MacAddress:ea:58:ec:d2:bf:3d Speed:10000 Mtu:8901} {Name:2bd9779bc021cfb MacAddress:ea:79:06:ff:11:2f Speed:10000 Mtu:8901} {Name:2ea91fd186a7fcf MacAddress:86:3a:79:2c:a6:c7 Speed:10000 Mtu:8901} {Name:4b594053a82b29c MacAddress:3e:53:03:05:35:f4 Speed:10000 Mtu:8901} {Name:5c86d6fabe71450 MacAddress:a2:ce:14:73:14:60 Speed:10000 Mtu:8901} {Name:860aa432073c29c MacAddress:3a:f2:e2:e3:ee:de Speed:10000 Mtu:8901} {Name:ae8882d35f39348 MacAddress:6e:bf:51:24:00:d3 Speed:10000 Mtu:8901} {Name:b2884df347214fc MacAddress:d2:d2:74:78:12:b1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:a4:e1:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:3e:29:19:fb:32 Speed:0 Mtu:8901} {Name:cfbe8234d4bbc93 MacAddress:fe:a1:a7:9c:48:85 Speed:10000 Mtu:8901} {Name:d708773746750d0 MacAddress:32:48:f6:e0:a6:ba Speed:10000 Mtu:8901} {Name:ed105230a355977 MacAddress:92:d8:1e:4a:5c:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:a4:e1:9f Speed:0 Mtu:9001} {Name:ff5e186750cb7c4 MacAddress:46:00:73:26:72:21 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ee:f6:0a:98:59:4c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:cc:13:99:42:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:19:54.089363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089097 6907 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:19:54.089363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089146 6907 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:19:54.089430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089403 6907 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:19:54.089577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089428 6907 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-153.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:19:54.089619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089579 6907 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:19:54.089619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089588 6907 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:19:54.089619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089600 6907 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:19:54.089699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089621 6907 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:19:54.089730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089725 6907 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:19:54.089895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089885 6907 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:19:54.090000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.089992 6907 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:19:54.090030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.090027 6907 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:19:54.090057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.090037 6907 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:19:54.090057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.090048 6907 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:19:54.090057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.090057 6907 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:19:54.090193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.090067 6907 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:19:54.091027 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.091012 6907 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:19:54.091070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.091028 6907 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:19:54.091634 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.091613 6907 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:19:54.094260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094246 6907 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:19:54.094309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094260 6907 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:19:54.094371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094361 6907 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:19:54.094576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094565 6907 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:19:54.094609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094594 6907 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:19:54.094694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094685 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:19:54.094727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094699 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:19:54.094727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094707 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:19:54.094727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094713 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:19:54.094727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094719 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:19:54.094727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094725 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094731 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094736 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094743 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094749 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094764 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094773 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094807 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:19:54.094859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.094817 6907 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:19:54.099367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.099347 6907 server.go:1263] "Started kubelet" Oct 30 19:19:54.099497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.099420 6907 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:19:54.099553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.099498 6907 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:19:54.099601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.099573 6907 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:19:54.099978 ip-10-0-132-153 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:19:54.101001 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.100965 6907 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:19:54.101221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.101051 6907 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:19:54.103479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.103461 6907 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:19:54.103772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.103747 6907 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:19:54.104352 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.104318 6907 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-153.us-east-2.compute.internal\" not found" Oct 30 19:19:54.104453 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104390 6907 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:13:44 +0000 UTC" deadline="2028-04-08 14:36:01.981300927 +0000 UTC" Oct 30 19:19:54.104453 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104405 6907 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12619h16m7.876898313s" Oct 30 19:19:54.104683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104667 6907 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:19:54.104984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104967 6907 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:19:54.105060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104986 6907 factory.go:55] Registering systemd factory Oct 30 19:19:54.105060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.104996 6907 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:19:54.106147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.106113 6907 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:19:54.106147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.106137 6907 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:19:54.108282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.108259 6907 factory.go:153] Registering CRI-O factory Oct 30 19:19:54.108392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.108288 6907 factory.go:223] Registration of the crio container factory successfully Oct 30 19:19:54.108392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.108311 6907 factory.go:103] Registering Raw factory Oct 30 19:19:54.108392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.108326 6907 manager.go:1199] Started watching for new ooms in manager Oct 30 19:19:54.108870 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.108854 6907 manager.go:319] Starting recovery of all containers Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.113552 6907 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.114364 6907 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.114418 6907 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115671 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115724 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dcea2b7-e349-4e35-b71a-35ad85f61c54" volumeName="kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115738 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18" volumeName="kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115750 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115762 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115772 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4738657b-e0c8-4a35-9893-85f99f3632b2" volumeName="kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115785 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e5dcce8-8409-491b-8428-eecdaf1ea2c0" volumeName="kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115794 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115808 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115819 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d167a43-62ce-4e3b-98ad-a095c3276ea8" volumeName="kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115834 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115845 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115856 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115868 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115883 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115892 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364bcd9-a94f-4072-973c-bc3d9993d2bd" volumeName="kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115913 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115920 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115927 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.115924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115948 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23" volumeName="kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115959 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364bcd9-a94f-4072-973c-bc3d9993d2bd" volumeName="kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115969 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.115984 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116006 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116016 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b270d8-4b48-48c9-a930-99d9c35419cc" volumeName="kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116027 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" volumeName="kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116036 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116049 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b81ec2a-4636-47b1-93dd-c38322d944f7" volumeName="kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116063 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116086 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa53f8d4-8928-4045-b84f-9d92c83c66e2" volumeName="kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116099 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116109 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116123 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116133 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e5dcce8-8409-491b-8428-eecdaf1ea2c0" volumeName="kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116145 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d167a43-62ce-4e3b-98ad-a095c3276ea8" volumeName="kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116156 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f10dec5f-455a-4587-8c73-6f9c872a0372" volumeName="kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116165 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" volumeName="kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116178 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116190 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116201 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116209 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116220 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116231 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116242 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b270d8-4b48-48c9-a930-99d9c35419cc" volumeName="kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116252 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb207c2c-2bf8-4263-b349-aa2f5be7b0fb" volumeName="kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116275 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116286 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116298 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116308 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116322 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116336 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116348 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18" volumeName="kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116359 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116370 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116381 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39f57a2e-06f1-4918-a6b3-e93802969d3d" volumeName="kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116394 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d167a43-62ce-4e3b-98ad-a095c3276ea8" volumeName="kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116411 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c96d731e-41cf-442f-adfc-3d36a9908b31" volumeName="kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116421 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f10dec5f-455a-4587-8c73-6f9c872a0372" volumeName="kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116433 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa53f8d4-8928-4045-b84f-9d92c83c66e2" volumeName="kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116457 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116468 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116481 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116492 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23" volumeName="kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116503 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116519 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116529 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116536 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" volumeName="kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116547 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116561 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc2ecc8f-2f83-4ead-a40a-f3d487ba622b" volumeName="kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116573 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f10dec5f-455a-4587-8c73-6f9c872a0372" volumeName="kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116586 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116598 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116609 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116622 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116629 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e5dcce8-8409-491b-8428-eecdaf1ea2c0" volumeName="kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116636 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116643 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" volumeName="kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116650 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa53f8d4-8928-4045-b84f-9d92c83c66e2" volumeName="kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116658 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364bcd9-a94f-4072-973c-bc3d9993d2bd" volumeName="kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116667 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116674 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dcea2b7-e349-4e35-b71a-35ad85f61c54" volumeName="kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116681 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d167a43-62ce-4e3b-98ad-a095c3276ea8" volumeName="kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116689 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c96d731e-41cf-442f-adfc-3d36a9908b31" volumeName="kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116697 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb207c2c-2bf8-4263-b349-aa2f5be7b0fb" volumeName="kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116708 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116720 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="39f57a2e-06f1-4918-a6b3-e93802969d3d" volumeName="kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116728 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5364bcd9-a94f-4072-973c-bc3d9993d2bd" volumeName="kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116735 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116744 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa53f8d4-8928-4045-b84f-9d92c83c66e2" volumeName="kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116751 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116758 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116766 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b81ec2a-4636-47b1-93dd-c38322d944f7" volumeName="kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116776 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116787 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8dcea2b7-e349-4e35-b71a-35ad85f61c54" volumeName="kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116798 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" volumeName="kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116809 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" volumeName="kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116820 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01332dde-ec0a-41fd-abfc-f73fab46b857" volumeName="kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116833 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17c9150e-8869-4602-b9c3-57ed6659c8f6" volumeName="kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116844 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aaa7bfb-b228-4955-9670-bb51ae187fed" volumeName="kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116856 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116869 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="306fb251-67bf-402f-a8a4-2da67c1b2013" volumeName="kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116882 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116892 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f10dec5f-455a-4587-8c73-6f9c872a0372" volumeName="kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116905 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d3aaac-34ae-41ca-883b-f4b6e0bdd642" volumeName="kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116917 6907 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="625638da-1fc7-46f4-86f6-f87b66e4cc1f" volumeName="kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116927 6907 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:19:54.117055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.116933 6907 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:19:54.120671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.120124 6907 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:19:54.141388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.141354 6907 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:19:54.142738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.142716 6907 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:19:54.142738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.142733 6907 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:19:54.142913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.142753 6907 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:19:54.142913 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.142795 6907 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:19:54.146819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.146791 6907 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:19:54.151358 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.151339 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:54.152262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.152230 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:54.170559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170519 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="817238c930acd8196dd00a44f41f7b8856968b9e2c25b10f0b9af070e154126c" exitCode=0 Oct 30 19:19:54.170559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170556 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="e1e7194435d553a015cea68457760d28ba9abbee444bda27a9713a79c5e515be" exitCode=0 Oct 30 19:19:54.170729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170572 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="822d5e16e30f43cde1728faca4ba885e87f27c8ca4f0d4fdb11ea5f4c273d2d0" exitCode=0 Oct 30 19:19:54.170729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170584 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="cddbe5d123ce1cbfff79f70e6744beb9eecbecfa54b3de72bc3831fa2c83ab7a" exitCode=0 Oct 30 19:19:54.170729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170593 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="2552b9828dd4e2d158c5d0e56df30be0b213568c53122e0dd4a4e652561eefa7" exitCode=0 Oct 30 19:19:54.170729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.170604 6907 generic.go:356] "Generic (PLEG): container finished" podID="9d167a43-62ce-4e3b-98ad-a095c3276ea8" containerID="250c1b246fa4a50e6581de0715bc06808f2c4936f2739a6584e5636fb0b510f3" exitCode=0 Oct 30 19:19:54.174505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.174480 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:19:54.174862 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.174834 6907 generic.go:356] "Generic (PLEG): container finished" podID="f92253fd-8219-46e9-b7c6-a4ccebbd7dd4" containerID="06f0505eb1de34387f98b0f34f42b0ab369201452abc80e281b0e48720d93903" exitCode=1 Oct 30 19:19:54.185763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.185736 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" exitCode=0 Oct 30 19:19:54.188046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.188014 6907 generic.go:356] "Generic (PLEG): container finished" podID="2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0" containerID="e0e695210d242f29c42427a054c8c39336c5462dbadbba521da59696fb748707" exitCode=0 Oct 30 19:19:54.195510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.195485 6907 generic.go:356] "Generic (PLEG): container finished" podID="e9876e1c179ef0e136f2b6103bc70569" containerID="b5494c5f72d38ba5f60770882adca52ead2dc012c2af4463de94c2466190a458" exitCode=0 Oct 30 19:19:54.199266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.199241 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" exitCode=0 Oct 30 19:19:54.240755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.240733 6907 manager.go:324] Recovery completed Oct 30 19:19:54.242848 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.242829 6907 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:19:54.266658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266627 6907 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:19:54.266658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266646 6907 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:19:54.266833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266669 6907 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:19:54.266833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266799 6907 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:19:54.266833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266811 6907 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:19:54.266833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266828 6907 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:19:54.266833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266833 6907 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:19:54.267050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266838 6907 policy_none.go:50] "Start" Oct 30 19:19:54.267050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266848 6907 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:19:54.267050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.266857 6907 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:19:54.278224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.278198 6907 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:19:54.278224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.278222 6907 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:19:54.278224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.278230 6907 policy_none.go:44] "Start" Oct 30 19:19:54.283127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.283097 6907 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:19:54.286370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.286354 6907 manager.go:341] "Starting Device Plugin manager" Oct 30 19:19:54.287123 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.286387 6907 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:19:54.287178 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287128 6907 server.go:87] "Starting device plugin registration server" Oct 30 19:19:54.287466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287438 6907 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:19:54.287501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287470 6907 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:19:54.287569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287554 6907 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:19:54.287692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287656 6907 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:19:54.287749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287702 6907 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:19:54.287749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287710 6907 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:19:54.287851 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.287759 6907 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:19:54.287691073Z","UUID":"cac40b7a-9074-492b-b573-db60b8d1f4db","Handler":null,"Name":"","Endpoint":""} Oct 30 19:19:54.289578 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.289547 6907 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:19:54.289835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.289811 6907 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:19:54.289960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.289842 6907 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:19:54.387776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.387736 6907 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:19:54.390015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.389994 6907 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:19:54.390146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.390028 6907 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:19:54.390146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.390040 6907 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-153.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:19:54.390146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.390094 6907 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.405584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.405565 6907 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.405678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.405637 6907 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.443823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.443779 6907 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal"] Oct 30 19:19:54.444520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444505 6907 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.444575 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444535 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"1ae4921b5faf0892072f47ac728ceb26","Type":"ContainerStarted","Data":"621039d322aa10514228f85c9d5e30f49d52b9e8361c54e2c8afc067d96ee5cb"} Oct 30 19:19:54.444608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444507 6907 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.444608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444593 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"1ae4921b5faf0892072f47ac728ceb26","Type":"ContainerStarted","Data":"df435f0c88c7a1822c8113759bb72825c4792a24216f9b3fd9dc3ab40e4d90c7"} Oct 30 19:19:54.444693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444613 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerStarted","Data":"5f059f4047ae026e8af63827b24dd3c92c8e3b96d257a129e809b3a40e6563fa"} Oct 30 19:19:54.444693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444630 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerDied","Data":"b5494c5f72d38ba5f60770882adca52ead2dc012c2af4463de94c2466190a458"} Oct 30 19:19:54.444693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.444650 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" event={"ID":"e9876e1c179ef0e136f2b6103bc70569","Type":"ContainerStarted","Data":"8c275c88fc4a8d39126a320561fc4b68912182480c37c200b273ab2763a034da"} Oct 30 19:19:54.454324 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.454305 6907 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:54.454421 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.454355 6907 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.455359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.455346 6907 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:54.455402 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:54.455379 6907 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.521788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.521716 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.521788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.521749 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.521788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.521766 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622369 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622334 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622373 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622398 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622468 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622502 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e9876e1c179ef0e136f2b6103bc70569-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"e9876e1c179ef0e136f2b6103bc70569\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:54.622649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:54.622552 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ae4921b5faf0892072f47ac728ceb26-config\") pod \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" (UID: \"1ae4921b5faf0892072f47ac728ceb26\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:55.091165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.091129 6907 apiserver.go:52] "Watching apiserver" Oct 30 19:19:55.100241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.100216 6907 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:19:55.100904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.100877 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-h2bcc","openshift-dns/dns-default-dxxbw","openshift-multus/network-metrics-daemon-dmtjr","openshift-network-diagnostics/network-check-target-h9j28","openshift-insights/insights-runtime-extractor-fv898","openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz","openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6","openshift-monitoring/alertmanager-main-0","openshift-monitoring/monitoring-plugin-77f4f76594-snf6t","openshift-ingress-canary/ingress-canary-wrv7f","openshift-monitoring/node-exporter-p5rpc","kube-system/global-pull-secret-syncer-n84k8","kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-wv29b","openshift-image-registry/node-ca-cxnr9","kube-system/konnectivity-agent-h9jxz","openshift-dns/node-resolver-6kvp2","openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq","openshift-cluster-node-tuning-operator/tuned-x8548","openshift-multus/multus-additional-cni-plugins-cfdq9","openshift-network-operator/iptables-alerter-xlfrt"] Oct 30 19:19:55.106584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.106548 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.106584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.106583 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:19:55.106775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.106724 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.106775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.106726 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.106913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.106896 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:19:55.107024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.107003 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.107177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.107161 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.107640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.107622 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.107869 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.107851 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.108250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108218 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vd8dh\"" Oct 30 19:19:55.108351 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108313 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:19:55.108432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108387 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:19:55.108530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108458 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-t757m\"" Oct 30 19:19:55.108630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108612 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.108790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108776 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:19:55.108947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108918 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:19:55.109078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109056 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:19:55.109078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109068 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vj9wr\"" Oct 30 19:19:55.109189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109136 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.109189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109159 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n5j86\"" Oct 30 19:19:55.109290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109264 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:19:55.109336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.108778 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vhzsf\"" Oct 30 19:19:55.109380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109352 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.109472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.109439 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dk8pw\"" Oct 30 19:19:55.110128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110110 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:19:55.110214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110159 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.110342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110327 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:19:55.110392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110379 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t67x5\"" Oct 30 19:19:55.110476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110454 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.110583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.110560 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ctbj8\"" Oct 30 19:19:55.111413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.111394 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xs9gl\"" Oct 30 19:19:55.111582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.111559 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:19:55.111699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.111590 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4vgss\"" Oct 30 19:19:55.111699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.111656 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:19:55.112018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.112003 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:19:55.114048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114011 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.114048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114033 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.114194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114104 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gkcxf\"" Oct 30 19:19:55.114709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114694 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6rfl7\"" Oct 30 19:19:55.114917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114893 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.114992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114919 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhqgv\"" Oct 30 19:19:55.114992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114929 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.114992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114965 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:19:55.114992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.114973 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:19:55.115214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115086 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:19:55.115288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115258 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:19:55.115347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115284 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:19:55.115347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115283 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zxgfv\"" Oct 30 19:19:55.115347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115302 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.115347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115299 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.115347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115341 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:19:55.115601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115347 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:19:55.115601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115366 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.115601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115461 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.115601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.115561 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktbkj\"" Oct 30 19:19:55.116100 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116084 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:19:55.116378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116359 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:19:55.116496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116478 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:19:55.116656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116641 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:19:55.116712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116662 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7fl45\"" Oct 30 19:19:55.116712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116669 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:19:55.116779 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116716 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:19:55.116819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116796 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:19:55.116866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116819 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:19:55.116866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.116833 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j9nbc\"" Oct 30 19:19:55.117159 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117141 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:19:55.117260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117239 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9rkks\"" Oct 30 19:19:55.117396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117384 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:19:55.117548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117535 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:19:55.117923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117898 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:19:55.117994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.117942 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:19:55.118191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.118170 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:19:55.118260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.118171 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:19:55.118521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.118506 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:19:55.119607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.119594 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sz2qj\"" Oct 30 19:19:55.119680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.119622 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:19:55.119680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.119651 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:19:55.119737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.119689 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:19:55.120190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120174 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 19:19:55.120454 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120430 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:19:55.120728 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120712 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 19:19:55.120795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120734 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 19:19:55.120795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120773 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:19:55.120966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120950 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.120976 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qmmv4\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121011 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121014 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3cesc8eim54g8\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121026 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-svkv2\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121017 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:19:55.121038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121039 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121052 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121060 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121068 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j6fc8\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121050 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121065 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121140 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2kk4cgcauul0i\"" Oct 30 19:19:55.121256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.121197 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:19:55.124063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124050 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gcr25\"" Oct 30 19:19:55.124118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124048 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.124118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124091 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.124192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124118 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.124192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124138 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.124192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124143 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-tmp-dir\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.124323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124192 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.124323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124237 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.124674 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124652 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4738657b-e0c8-4a35-9893-85f99f3632b2-original-pull-secret\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.124818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124700 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.124818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124750 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.124937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124832 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkh7w\" (UniqueName: \"kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.124937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124864 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.124937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124870 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-binary-copy\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.124937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124914 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.125064 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124948 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.125064 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.124983 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.125186 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125165 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.125247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125177 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d167a43-62ce-4e3b-98ad-a095c3276ea8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.125247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125221 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:19:55.125339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125288 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.125339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125315 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.125464 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125354 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.125464 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125409 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz44v\" (UniqueName: \"kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.125560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125520 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-agent-certs\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:19:55.125593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125561 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125586 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.125694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125602 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.125694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125637 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125682 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125791 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125778 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125806 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cni-binary-copy\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125810 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cw9\" (UniqueName: \"kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:55.125938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125922 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.125987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125945 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.125987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125960 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.125984 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfv2\" (UniqueName: \"kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126014 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.126085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126070 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srnf\" (UniqueName: \"kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.126202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126097 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.126202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126114 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.126299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126243 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:19:55.126299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126275 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126300 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126325 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126351 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126379 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126408 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126411 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23-konnectivity-ca\") pod \"konnectivity-agent-h9jxz\" (UID: \"3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23\") " pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126473 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126541 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126564 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfcd\" (UniqueName: \"kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.126595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126586 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-daemon-config\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126601 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126627 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrpc\" (UniqueName: \"kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126650 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126674 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126760 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b81ec2a-4636-47b1-93dd-c38322d944f7-metrics-certs\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:55.126797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.126766 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c96d731e-41cf-442f-adfc-3d36a9908b31-iptables-alerter-script\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.127214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.127196 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-serviceca\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.132278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.132255 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz44v\" (UniqueName: \"kubernetes.io/projected/01332dde-ec0a-41fd-abfc-f73fab46b857-kube-api-access-zz44v\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.132363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.132287 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srnf\" (UniqueName: \"kubernetes.io/projected/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-kube-api-access-4srnf\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.132476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.132459 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkh7w\" (UniqueName: \"kubernetes.io/projected/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-kube-api-access-kkh7w\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.132517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.132490 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cw9\" (UniqueName: \"kubernetes.io/projected/3b81ec2a-4636-47b1-93dd-c38322d944f7-kube-api-access-j5cw9\") pod \"network-metrics-daemon-dmtjr\" (UID: \"3b81ec2a-4636-47b1-93dd-c38322d944f7\") " pod="openshift-multus/network-metrics-daemon-dmtjr" Oct 30 19:19:55.133285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.133263 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfv2\" (UniqueName: \"kubernetes.io/projected/8dcea2b7-e349-4e35-b71a-35ad85f61c54-kube-api-access-rtfv2\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.133381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.133315 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz6k\" (UniqueName: \"kubernetes.io/projected/dc2ecc8f-2f83-4ead-a40a-f3d487ba622b-kube-api-access-vjz6k\") pod \"network-check-target-h9j28\" (UID: \"dc2ecc8f-2f83-4ead-a40a-f3d487ba622b\") " pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:19:55.133689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.133670 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfcd\" (UniqueName: \"kubernetes.io/projected/9d167a43-62ce-4e3b-98ad-a095c3276ea8-kube-api-access-fqfcd\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.133909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.133889 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrpc\" (UniqueName: \"kubernetes.io/projected/c96d731e-41cf-442f-adfc-3d36a9908b31-kube-api-access-2jrpc\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.206165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.206130 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:19:55.206165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.206131 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:19:55.206363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.206303 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:19:55.206898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.206875 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:19:55.207071 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207056 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:19:55.207130 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207060 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:19:55.207346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207329 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:19:55.207418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207362 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8dg65acfiq0rj\"" Oct 30 19:19:55.207418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207390 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:19:55.207546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207533 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:19:55.207816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.207802 6907 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:19:55.208151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.208134 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:19:55.208280 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.208266 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:19:55.227062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227028 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.227176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227071 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.227176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227099 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.227176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227122 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.227176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227146 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.227176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227170 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227191 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227202 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-hostroot\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227196 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227173 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-socket-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227255 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bxs\" (UniqueName: \"kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227283 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227310 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227316 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227335 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227364 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227383 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/306fb251-67bf-402f-a8a4-2da67c1b2013-metrics-client-ca\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227389 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227408 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306fb251-67bf-402f-a8a4-2da67c1b2013-tmp\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.227426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227417 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-config\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227435 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227474 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227477 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227492 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227521 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227545 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5364bcd9-a94f-4072-973c-bc3d9993d2bd-tmp-dir\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227551 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227568 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227581 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8h8\" (UniqueName: \"kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227608 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227627 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-netns\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227632 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227660 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227686 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227697 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227710 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227707 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227742 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227786 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-sys-fs\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227792 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227800 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227814 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-textfile\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227711 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227830 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227839 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227842 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227872 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227907 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227961 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.227977 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-kubelet-config\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228015 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.228069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228064 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228097 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5scm\" (UniqueName: \"kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228121 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228131 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228148 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228177 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228188 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-multus-certs\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228204 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228233 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228246 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb207c2c-2bf8-4263-b349-aa2f5be7b0fb-hosts-file\") pod \"node-resolver-6kvp2\" (UID: \"cb207c2c-2bf8-4263-b349-aa2f5be7b0fb\") " pod="openshift-dns/node-resolver-6kvp2" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228259 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7w2\" (UniqueName: \"kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228285 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228309 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228312 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-tls\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228324 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228333 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228357 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228368 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228396 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228430 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs26\" (UniqueName: \"kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228493 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228523 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228549 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228550 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228571 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228575 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228619 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-bin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228621 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228639 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-run-k8s-cni-cncf-io\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228645 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2aaa7bfb-b228-4955-9670-bb51ae187fed-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228661 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228675 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-cnibin\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228691 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228718 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228745 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4vg\" (UniqueName: \"kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228772 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228801 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228831 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228853 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-client-certs\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228861 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228868 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228899 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-tmp\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.228334 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229061 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229089 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229130 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglxr\" (UniqueName: \"kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229155 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229181 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229207 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229218 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-grpc-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229230 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229276 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229302 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229309 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229327 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229369 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.229368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229342 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229432 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-cnibin\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229485 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229514 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229523 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f10dec5f-455a-4587-8c73-6f9c872a0372-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229541 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229582 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229576 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229656 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229670 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-system-cni-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229704 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229755 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229778 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229804 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-metrics-server-audit-profiles\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229810 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhlq\" (UniqueName: \"kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229853 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229879 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.229979 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230010 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230038 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230065 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7p6t\" (UniqueName: \"kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230091 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230121 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230131 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovn-node-metrics-cert\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230149 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230178 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-ovnkube-script-lib\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230190 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230220 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230321 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230346 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230353 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230363 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5364bcd9-a94f-4072-973c-bc3d9993d2bd-metrics-tls\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230382 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230398 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-socket-dir-parent\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230409 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230479 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230501 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230516 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230531 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230548 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230564 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cwd\" (UniqueName: \"kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230579 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230605 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230621 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-tuned\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230620 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230665 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c96d731e-41cf-442f-adfc-3d36a9908b31-host-slash\") pod \"iptables-alerter-xlfrt\" (UID: \"c96d731e-41cf-442f-adfc-3d36a9908b31\") " pod="openshift-network-operator/iptables-alerter-xlfrt" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230673 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230819 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230850 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-metrics-client-ca\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230878 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230878 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230908 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230944 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4738657b-e0c8-4a35-9893-85f99f3632b2-dbus\") pod \"global-pull-secret-syncer-n84k8\" (UID: \"4738657b-e0c8-4a35-9893-85f99f3632b2\") " pod="kube-system/global-pull-secret-syncer-n84k8" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230964 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.230966 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-env-overrides\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231005 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa53f8d4-8928-4045-b84f-9d92c83c66e2-data-volume\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231011 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231038 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231047 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f10dec5f-455a-4587-8c73-6f9c872a0372-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231039 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231065 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d167a43-62ce-4e3b-98ad-a095c3276ea8-system-cni-dir\") pod \"multus-additional-cni-plugins-cfdq9\" (UID: \"9d167a43-62ce-4e3b-98ad-a095c3276ea8\") " pod="openshift-multus/multus-additional-cni-plugins-cfdq9" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231099 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231533 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231584 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231613 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231642 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-tls\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231647 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzwt\" (UniqueName: \"kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231738 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.231819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231810 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231845 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231962 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.231996 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232026 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232085 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-multus-conf-dir\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232085 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232094 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fa53f8d4-8928-4045-b84f-9d92c83c66e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232134 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232170 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232199 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232202 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232320 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/306fb251-67bf-402f-a8a4-2da67c1b2013-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232331 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232432 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232534 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39f57a2e-06f1-4918-a6b3-e93802969d3d-cert\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232550 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232590 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232627 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232632 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-cni-multus\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232659 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232665 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-secret-metrics-server-tls\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232689 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232724 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232758 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232793 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232827 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.232830 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233027 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-device-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233068 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233087 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233100 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233113 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-client-ca-bundle\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233159 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233169 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233235 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233381 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233387 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233406 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5364bcd9-a94f-4072-973c-bc3d9993d2bd-config-volume\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233423 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233667 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233723 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233775 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.233979 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/92b270d8-4b48-48c9-a930-99d9c35419cc-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234033 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234093 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234125 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234141 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234177 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234210 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234464 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234499 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234527 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234563 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234598 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234666 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234707 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234712 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234742 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.234789 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-etc-kubernetes\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235025 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-audit-log\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235071 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.235094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235077 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dcea2b7-e349-4e35-b71a-35ad85f61c54-host-var-lib-kubelet\") pod \"multus-h2bcc\" (UID: \"8dcea2b7-e349-4e35-b71a-35ad85f61c54\") " pod="openshift-multus/multus-h2bcc" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235147 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18-host\") pod \"node-ca-cxnr9\" (UID: \"a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18\") " pod="openshift-image-registry/node-ca-cxnr9" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235229 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8h8\" (UniqueName: \"kubernetes.io/projected/39f57a2e-06f1-4918-a6b3-e93802969d3d-kube-api-access-mc8h8\") pod \"ingress-canary-wrv7f\" (UID: \"39f57a2e-06f1-4918-a6b3-e93802969d3d\") " pod="openshift-ingress-canary/ingress-canary-wrv7f" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235302 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/01332dde-ec0a-41fd-abfc-f73fab46b857-registration-dir\") pod \"aws-ebs-csi-driver-node-tzlm8\" (UID: \"01332dde-ec0a-41fd-abfc-f73fab46b857\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzlm8" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235334 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bxs\" (UniqueName: \"kubernetes.io/projected/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-kube-api-access-84bxs\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235383 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.235769 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs26\" (UniqueName: \"kubernetes.io/projected/5364bcd9-a94f-4072-973c-bc3d9993d2bd-kube-api-access-7qs26\") pod \"dns-default-dxxbw\" (UID: \"5364bcd9-a94f-4072-973c-bc3d9993d2bd\") " pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.236095 6907 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.236222 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5scm\" (UniqueName: \"kubernetes.io/projected/2aaa7bfb-b228-4955-9670-bb51ae187fed-kube-api-access-j5scm\") pod \"kube-state-metrics-5dc8d7d89b-xwhmz\" (UID: \"2aaa7bfb-b228-4955-9670-bb51ae187fed\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-xwhmz" Oct 30 19:19:55.237790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.236552 6907 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:55.238214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238201 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhlq\" (UniqueName: \"kubernetes.io/projected/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-kube-api-access-4mhlq\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.238468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238424 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7w2\" (UniqueName: \"kubernetes.io/projected/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-kube-api-access-xc7w2\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.238550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238427 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglxr\" (UniqueName: \"kubernetes.io/projected/f10dec5f-455a-4587-8c73-6f9c872a0372-kube-api-access-wglxr\") pod \"openshift-state-metrics-7bd7489957-8g6jq\" (UID: \"f10dec5f-455a-4587-8c73-6f9c872a0372\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8g6jq" Oct 30 19:19:55.238896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238874 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4vg\" (UniqueName: \"kubernetes.io/projected/306fb251-67bf-402f-a8a4-2da67c1b2013-kube-api-access-9b4vg\") pod \"thanos-querier-7dff9b6c5c-ghxcg\" (UID: \"306fb251-67bf-402f-a8a4-2da67c1b2013\") " pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:55.238960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238891 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cwd\" (UniqueName: \"kubernetes.io/projected/92b270d8-4b48-48c9-a930-99d9c35419cc-kube-api-access-q9cwd\") pod \"monitoring-plugin-77f4f76594-snf6t\" (UID: \"92b270d8-4b48-48c9-a930-99d9c35419cc\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:55.238960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238900 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"alertmanager-main-0\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:19:55.238960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238896 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7p6t\" (UniqueName: \"kubernetes.io/projected/33d3aaac-34ae-41ca-883b-f4b6e0bdd642-kube-api-access-b7p6t\") pod \"metrics-server-6c5b8d6bd7-rp9k6\" (UID: \"33d3aaac-34ae-41ca-883b-f4b6e0bdd642\") " pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:19:55.238960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.238951 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzwt\" (UniqueName: \"kubernetes.io/projected/fa53f8d4-8928-4045-b84f-9d92c83c66e2-kube-api-access-9nzwt\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.239378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.239362 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"prometheus-k8s-0\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:55.247758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.247737 6907 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:55.247848 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:55.247789 6907 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:55.248502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.248484 6907 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:19:55.248566 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:19:55.248526 6907 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-153.us-east-2.compute.internal" Oct 30 19:19:55.335749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335718 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335751 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335774 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335791 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335808 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335848 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-node-log\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335868 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-netns\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335859 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-var-lib-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335866 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-root\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335897 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335871 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-run\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.335972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335952 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335989 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.335996 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336019 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336038 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-conf\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336041 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336068 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336069 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-netd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336089 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336123 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-slash\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336133 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-cni-bin\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336127 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysctl-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336158 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336125 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-host\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336204 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336205 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-lib-modules\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336232 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336251 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fa53f8d4-8928-4045-b84f-9d92c83c66e2-crio-socket\") pod \"insights-runtime-extractor-fv898\" (UID: \"fa53f8d4-8928-4045-b84f-9d92c83c66e2\") " pod="openshift-insights/insights-runtime-extractor-fv898" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336264 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336295 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336319 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336322 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-etc-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336325 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-systemd\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.336359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336345 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336379 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336350 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-sys\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336400 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-kubelet\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336417 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336431 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-openvswitch\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336458 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336463 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-log-socket\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336468 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-kubernetes\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336488 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336512 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336532 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-sys\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336539 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336553 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-systemd\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336564 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336580 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336581 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-systemd-units\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336598 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-modprobe-d\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336643 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336653 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-etc-sysconfig\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336661 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336681 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f92253fd-8219-46e9-b7c6-a4ccebbd7dd4-run-ovn\") pod \"ovnkube-node-wv29b\" (UID: \"f92253fd-8219-46e9-b7c6-a4ccebbd7dd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336687 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e5dcce8-8409-491b-8428-eecdaf1ea2c0-var-lib-kubelet\") pod \"tuned-x8548\" (UID: \"4e5dcce8-8409-491b-8428-eecdaf1ea2c0\") " pod="openshift-cluster-node-tuning-operator/tuned-x8548" Oct 30 19:19:55.337070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.336776 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0-node-exporter-wtmp\") pod \"node-exporter-p5rpc\" (UID: \"2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0\") " pod="openshift-monitoring/node-exporter-p5rpc" Oct 30 19:19:55.870830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:55.870800 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:19:56.550336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:56.550308 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dff9b6c5c-ghxcg" Oct 30 19:19:57.602702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:57.602667 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-snf6t" Oct 30 19:19:58.126519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:58.126480 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dxxbw" Oct 30 19:19:58.689475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:58.689430 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:19:59.083350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:19:59.083317 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h9j28" Oct 30 19:20:01.906424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:01.906394 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:20:01.906939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:01.906629 6907 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:20:07.734112 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:07.734069 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:20:12.008088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:12.008051 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wv29b" Oct 30 19:20:12.152079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:12.152049 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6c5b8d6bd7-rp9k6" Oct 30 19:20:17.625336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625298 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:17.625976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625876 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="config-reloader" containerID="cri-o://9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" gracePeriod=120 Oct 30 19:20:17.625976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625878 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-metric" containerID="cri-o://2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" gracePeriod=120 Oct 30 19:20:17.625976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625941 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy" containerID="cri-o://f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" gracePeriod=120 Oct 30 19:20:17.625976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625889 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-web" containerID="cri-o://c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" gracePeriod=120 Oct 30 19:20:17.626246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.626017 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="prom-label-proxy" containerID="cri-o://cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" gracePeriod=120 Oct 30 19:20:17.626246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.625813 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="alertmanager" containerID="cri-o://f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" gracePeriod=120 Oct 30 19:20:17.875502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:17.875420 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.041770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041723 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.041770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041769 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041791 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041813 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041834 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.041939 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042001 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042032 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042062 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042097 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042156 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042209 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042241 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") pod \"17c9150e-8869-4602-b9c3-57ed6659c8f6\" (UID: \"17c9150e-8869-4602-b9c3-57ed6659c8f6\") " Oct 30 19:20:18.042898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.042867 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:18.043080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.043037 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:18.043817 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.043786 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:18.045954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.045922 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.045954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.045936 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "kube-api-access-dfbrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:18.046322 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046287 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:18.046422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046361 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.046422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046381 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:18.046551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046456 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.046551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046526 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.046716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.046693 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.049856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.049830 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.056980 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.056960 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config" pod "17c9150e-8869-4602-b9c3-57ed6659c8f6" (UID: "17c9150e-8869-4602-b9c3-57ed6659c8f6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143779 6907 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-main-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143812 6907 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143823 6907 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-cluster-tls-config\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143835 6907 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143845 6907 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143854 6907 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143864 6907 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-tls-assets\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.143867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143873 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17c9150e-8869-4602-b9c3-57ed6659c8f6-metrics-client-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.144214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143885 6907 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.144214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143894 6907 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17c9150e-8869-4602-b9c3-57ed6659c8f6-web-config\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.144214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143902 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbrh\" (UniqueName: \"kubernetes.io/projected/17c9150e-8869-4602-b9c3-57ed6659c8f6-kube-api-access-dfbrh\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.144214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143910 6907 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-config-out\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.144214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.143918 6907 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17c9150e-8869-4602-b9c3-57ed6659c8f6-alertmanager-main-db\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:18.323095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323061 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" exitCode=0 Oct 30 19:20:18.323095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323088 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" exitCode=0 Oct 30 19:20:18.323095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323096 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" exitCode=0 Oct 30 19:20:18.323095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323104 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" exitCode=0 Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323112 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" exitCode=0 Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323121 6907 generic.go:356] "Generic (PLEG): container finished" podID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" exitCode=0 Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323141 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323211 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323156 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323240 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323265 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323291 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323321 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} Oct 30 19:20:18.323354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323343 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17c9150e-8869-4602-b9c3-57ed6659c8f6","Type":"ContainerDied","Data":"5c86d6fabe71450ebd75b19c89d88c1f895ea8d2cb36def75b290d22b46819ae"} Oct 30 19:20:18.323805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.323293 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.331836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.331820 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.339289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.339273 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.345720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.345691 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:18.347467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.347436 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.353238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.353217 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:18.354517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.354496 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.361626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.361602 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.368720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.368701 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.375834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.375814 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.376123 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.376102 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.376199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376137 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.376199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376184 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.376425 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.376407 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.376487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376433 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.376487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376465 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.376684 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.376666 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.376723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376688 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.376723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376701 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.376913 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.376896 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.376971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376922 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.376971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.376941 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.377163 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.377148 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.377203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377166 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.377203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377179 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.377358 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.377344 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.377395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377360 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.377395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377372 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.377587 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:18.377570 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.377650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377595 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.377650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377613 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.377820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377803 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.377865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377821 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.378013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.377995 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.378081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378013 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.378224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378208 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.378275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378224 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.378419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378400 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.378484 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378419 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.378696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378674 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.378696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378695 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.378924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378908 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.378966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.378924 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.379108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379094 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.379156 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379109 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.379288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379275 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.379326 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379287 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.379512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379490 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.379512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379511 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.379784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379766 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.379784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379783 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.379995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379977 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.380050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.379996 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.380238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380221 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.380291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380239 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.380505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380483 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.380505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380505 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.380753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380736 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.380820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380765 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.380992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380976 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.381038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.380992 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.381194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381176 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.381268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381195 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.381380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381360 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.381424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381380 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.381576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381556 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.381649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381577 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.381783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381764 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.381829 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381783 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.381958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381941 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.381996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.381959 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.382175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.382153 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.382175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.382173 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.383402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.382557 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.383526 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.383407 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.383736 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.383718 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.383799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.383736 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.383979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.383961 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.384053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.383980 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.384281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384053 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:18.384281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384219 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.384281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384233 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.384422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384360 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="alertmanager" Oct 30 19:20:18.384422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384379 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="alertmanager" Oct 30 19:20:18.384422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384401 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="prom-label-proxy" Oct 30 19:20:18.384422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384410 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="prom-label-proxy" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384426 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="config-reloader" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384431 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="config-reloader" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384429 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384471 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384455 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384515 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384526 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="init-config-reloader" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384532 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="init-config-reloader" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384562 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-metric" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384567 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-metric" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384577 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-web" Oct 30 19:20:18.384580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384582 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-web" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384663 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-metric" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384675 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy-web" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384680 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="alertmanager" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384686 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="prom-label-proxy" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384693 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="config-reloader" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384699 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="kube-rbac-proxy" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384703 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" containerName="init-config-reloader" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384677 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.384964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384761 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.385240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384960 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.385240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.384976 6907 scope.go:122] "RemoveContainer" containerID="cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570" Oct 30 19:20:18.385240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385158 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570"} err="failed to get container status \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": rpc error: code = NotFound desc = could not find container \"cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570\": container with ID starting with cd337bc6f6767a510aec91aa9e41c94c8c9360cdae4761fcf56ded4d4d11f570 not found: ID does not exist" Oct 30 19:20:18.385240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385171 6907 scope.go:122] "RemoveContainer" containerID="2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569" Oct 30 19:20:18.385421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385406 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569"} err="failed to get container status \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": rpc error: code = NotFound desc = could not find container \"2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569\": container with ID starting with 2dcb145b228aeaf3d755d1c0c8ed35da1c8bb364f0e791bada33f7c5cfb15569 not found: ID does not exist" Oct 30 19:20:18.385479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385421 6907 scope.go:122] "RemoveContainer" containerID="f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813" Oct 30 19:20:18.385662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385642 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813"} err="failed to get container status \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": rpc error: code = NotFound desc = could not find container \"f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813\": container with ID starting with f2b0c2cc8a79c2ab3d4320394892bd726d16ac1cdd8dc753295a236b8ffd8813 not found: ID does not exist" Oct 30 19:20:18.385662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385661 6907 scope.go:122] "RemoveContainer" containerID="c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8" Oct 30 19:20:18.385813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385798 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8"} err="failed to get container status \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": rpc error: code = NotFound desc = could not find container \"c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8\": container with ID starting with c3dc89dbf700ce30fe76a4c63d85f4e5c8150c899aca4c731f18fdcc3a6f8df8 not found: ID does not exist" Oct 30 19:20:18.385852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385813 6907 scope.go:122] "RemoveContainer" containerID="9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306" Oct 30 19:20:18.385980 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385964 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306"} err="failed to get container status \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": rpc error: code = NotFound desc = could not find container \"9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306\": container with ID starting with 9585ff18b9a87eba83602fbc2daa4734d89e9a7d29bbc3c98fd9df52f26a9306 not found: ID does not exist" Oct 30 19:20:18.386021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.385980 6907 scope.go:122] "RemoveContainer" containerID="f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d" Oct 30 19:20:18.386173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.386155 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d"} err="failed to get container status \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": rpc error: code = NotFound desc = could not find container \"f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d\": container with ID starting with f9119cff14e5eceb425f8614f14fbd94d62ed8ec743a2d5ada0eeaf81220958d not found: ID does not exist" Oct 30 19:20:18.386216 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.386173 6907 scope.go:122] "RemoveContainer" containerID="75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4" Oct 30 19:20:18.386359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.386340 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4"} err="failed to get container status \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": rpc error: code = NotFound desc = could not find container \"75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4\": container with ID starting with 75b0467526b2176040cb3ff59da753d10f8f9e8bacc4601359ae110b4a0fe0d4 not found: ID does not exist" Oct 30 19:20:18.388663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.388647 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.394201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394115 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394332 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394345 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394393 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394407 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394393 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394438 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-sz2qj\"" Oct 30 19:20:18.394518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394484 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:20:18.395035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.394630 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:20:18.395822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.395806 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:20:18.406280 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.406250 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:18.547069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547027 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547066 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547097 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547139 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547183 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547330 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547372 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547420 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547560 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-config-out\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547595 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655bb\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-kube-api-access-655bb\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547661 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547684 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-web-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.547712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.547699 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648199 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655bb\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-kube-api-access-655bb\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648240 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648257 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-web-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648305 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648344 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648364 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648396 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648414 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648433 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648484 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648510 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648555 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648633 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-config-out\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.648805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648663 6907 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:20:18.649402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.648906 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.649490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.649400 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.650030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.649717 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20049954-3de4-452c-a17d-351680769c2e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.651248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.651221 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20049954-3de4-452c-a17d-351680769c2e-config-out\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.651570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.651544 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.651902 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.651771 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.651902 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.651784 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.652565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.651959 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-config-volume\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.652565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.652068 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.652565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.652226 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-web-config\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.652565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.652388 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20049954-3de4-452c-a17d-351680769c2e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.653381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.653365 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.657061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.657038 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655bb\" (UniqueName: \"kubernetes.io/projected/20049954-3de4-452c-a17d-351680769c2e-kube-api-access-655bb\") pod \"alertmanager-main-0\" (UID: \"20049954-3de4-452c-a17d-351680769c2e\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.708158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.708114 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:20:18.989663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:18.989627 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:20:18.993417 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:20:18.993381 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20049954_3de4_452c_a17d_351680769c2e.slice/crio-ac359b11f8bc9ef9859a8ce3c44a2a784a51e7b09a34a7912e22938fc303f6f6 WatchSource:0}: Error finding container ac359b11f8bc9ef9859a8ce3c44a2a784a51e7b09a34a7912e22938fc303f6f6: Status 404 returned error can't find the container with id ac359b11f8bc9ef9859a8ce3c44a2a784a51e7b09a34a7912e22938fc303f6f6 Oct 30 19:20:19.328463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:19.328365 6907 generic.go:356] "Generic (PLEG): container finished" podID="20049954-3de4-452c-a17d-351680769c2e" containerID="b0fb8490ff006b39a9e5486a3cb8a622c833897c383c2481d844f68b3d0273ac" exitCode=0 Oct 30 19:20:19.328627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:19.328470 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerDied","Data":"b0fb8490ff006b39a9e5486a3cb8a622c833897c383c2481d844f68b3d0273ac"} Oct 30 19:20:19.328627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:19.328549 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"ac359b11f8bc9ef9859a8ce3c44a2a784a51e7b09a34a7912e22938fc303f6f6"} Oct 30 19:20:20.148324 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.148290 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c9150e-8869-4602-b9c3-57ed6659c8f6" path="/var/lib/kubelet/pods/17c9150e-8869-4602-b9c3-57ed6659c8f6/volumes" Oct 30 19:20:20.336197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336158 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"8ab4dd7cf5185e594569d694e22a3a579619eb7489db457d31c293fbc3e15546"} Oct 30 19:20:20.336364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336241 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"e88bede7dc82c1dbc2a359b360fe12ebce4657923206bb0fc9d02f4e4ced0c0d"} Oct 30 19:20:20.336364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336279 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"5c13b6f9dbc5423d4b0a79705aee7f0874145efbd0be9fa9cb9429c463c270af"} Oct 30 19:20:20.336364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336317 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"c76042d018c9e9e6e4326393e37f282ef91e47d7a8004c1e237c6e081b40db06"} Oct 30 19:20:20.336364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336353 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"4e93b42f8a18a8fd3662132e443716691769a46e1a7a3d57d9f5bec45b0d3555"} Oct 30 19:20:20.336515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.336389 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20049954-3de4-452c-a17d-351680769c2e","Type":"ContainerStarted","Data":"0ecfa066a6e67d8dd0eade2c8a05202dedb64faccf121d81389d2955bbd91852"} Oct 30 19:20:20.363011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.362950 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.350320602 podStartE2EDuration="2.362931101s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.610499ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:20.361395011 +0000 UTC m=+26.337620948" watchObservedRunningTime="2026-10-30 19:20:20.362931101 +0000 UTC m=+26.339157045" Oct 30 19:20:20.748157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:20.748127 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h9jxz" Oct 30 19:20:21.944935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.944892 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:21.945381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945181 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="prometheus" containerID="cri-o://0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" gracePeriod=600 Oct 30 19:20:21.945381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945199 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy" containerID="cri-o://7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" gracePeriod=600 Oct 30 19:20:21.945381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945220 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="thanos-sidecar" containerID="cri-o://58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" gracePeriod=600 Oct 30 19:20:21.945732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945665 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" gracePeriod=600 Oct 30 19:20:21.946133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945799 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-web" containerID="cri-o://d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" gracePeriod=600 Oct 30 19:20:21.946133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:21.945838 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="config-reloader" containerID="cri-o://82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" gracePeriod=600 Oct 30 19:20:22.178082 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.178055 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.282840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282810 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.282840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282843 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282874 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282906 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282927 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282952 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.282993 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283219 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.283372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283316 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283381 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283438 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.283513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283461 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283478 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.283685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283526 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283736 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283703 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283736 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283768 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283806 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283836 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283865 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.283898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283892 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.284110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283924 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.284110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.283978 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") pod \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\" (UID: \"625638da-1fc7-46f4-86f6-f87b66e4cc1f\") " Oct 30 19:20:22.284530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.284238 6907 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.284530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.284264 6907 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.284530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.284280 6907 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.284530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.284373 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:22.284844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.284817 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.285597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.285567 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.286236 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286134 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "kube-api-access-5qb88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:22.286475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286416 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.286731 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286579 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.286832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286802 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:22.286920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286899 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:22.286978 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.286965 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.287328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.287294 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.287509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.287480 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:22.287771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.287751 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.287912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.287890 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.288174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.288155 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.288792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.288770 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.288996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.288979 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.297153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.297128 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config" pod "625638da-1fc7-46f4-86f6-f87b66e4cc1f" (UID: "625638da-1fc7-46f4-86f6-f87b66e4cc1f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:22.347452 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347415 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" exitCode=0 Oct 30 19:20:22.347452 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347457 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" exitCode=0 Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347465 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" exitCode=0 Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347471 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" exitCode=0 Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347476 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" exitCode=0 Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347481 6907 generic.go:356] "Generic (PLEG): container finished" podID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" exitCode=0 Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347495 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347521 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347564 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347591 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347616 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} Oct 30 19:20:22.347623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347628 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.347989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347642 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} Oct 30 19:20:22.347989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347666 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} Oct 30 19:20:22.347989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.347689 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"625638da-1fc7-46f4-86f6-f87b66e4cc1f","Type":"ContainerDied","Data":"2ea91fd186a7fcf93b0f8d077216bf5392d8eef6385a6389d5e752ccb90039ba"} Oct 30 19:20:22.355953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.355930 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.362815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.362799 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.369626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.369609 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.373430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.373400 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:22.377344 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.377321 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:22.377750 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.377735 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.385367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385344 6907 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-web-config\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385370 6907 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385383 6907 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385393 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qb88\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-kube-api-access-5qb88\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385410 6907 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/625638da-1fc7-46f4-86f6-f87b66e4cc1f-tls-assets\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385419 6907 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-db\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385428 6907 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385437 6907 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-kube-rbac-proxy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385466 6907 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config-out\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385478 6907 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385493 6907 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-grpc-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385508 6907 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385523 6907 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385538 6907 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/625638da-1fc7-46f4-86f6-f87b66e4cc1f-configmap-metrics-client-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385554 6907 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385568 6907 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-config\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.385748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.385583 6907 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/625638da-1fc7-46f4-86f6-f87b66e4cc1f-secret-metrics-client-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:22.386843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.386826 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.394433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.394414 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.401719 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.401700 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.401996 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.401980 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.402046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402006 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.402046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402024 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.402271 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.402251 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.402333 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402280 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.402333 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402303 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.402545 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.402528 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.402582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402549 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.402582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402562 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.402766 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.402751 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.402807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402769 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.402807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.402783 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.402999 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.402984 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.402999 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403001 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.403161 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403012 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.403283 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.403214 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.403283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403235 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.403283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403255 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.403632 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:22.403605 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.403689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403632 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.403689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403652 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.403970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403938 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.403970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.403970 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.404203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404180 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:22.404327 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404306 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.404552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404330 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.404636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404616 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.404693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404636 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.404693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404641 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-thanos" Oct 30 19:20:22.404693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404657 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-thanos" Oct 30 19:20:22.404693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404672 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy" Oct 30 19:20:22.404693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404680 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404711 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="init-config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404718 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="init-config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404732 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="prometheus" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404740 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="prometheus" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404754 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-web" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404760 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-web" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404776 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="thanos-sidecar" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404783 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="thanos-sidecar" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404796 6907 cpu_manager.go:414] "containerMap: removing container" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404800 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404856 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404866 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-thanos" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404874 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404888 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="init-config-reloader" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404894 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="prometheus" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404899 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="kube-rbac-proxy-web" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404905 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" containerName="thanos-sidecar" Oct 30 19:20:22.404912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404903 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.405500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.404930 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.405500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405134 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.405500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405148 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.405500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405339 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.405500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405354 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.405678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405577 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.405678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405591 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.405789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405773 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.405832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405789 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.406031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.405996 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.406031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406018 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.406296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406243 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.406296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406262 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.406542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406518 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.406542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406535 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.406805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406784 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.406864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.406806 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.407032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407015 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.407078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407033 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.407243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407225 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.407281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407243 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.407433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407416 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.407512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407434 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.407783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407764 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.407826 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407784 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.407998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407975 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.408044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.407998 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.408218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408202 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.408254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408219 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.408423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408408 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.408480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408424 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.408662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408646 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.408662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408661 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.408857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408840 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.408905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408871 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.408905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.408887 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.409175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409151 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.409218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409178 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.409476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409435 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.409552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409477 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.409739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409719 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.409739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409736 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.409970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409952 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.409970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.409969 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.410194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410179 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.410194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410193 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.410384 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410368 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.410384 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410383 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.410627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410610 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.410671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410627 6907 scope.go:122] "RemoveContainer" containerID="6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb" Oct 30 19:20:22.410816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410800 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb"} err="failed to get container status \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": rpc error: code = NotFound desc = could not find container \"6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb\": container with ID starting with 6fe02503f6130e2ab36d3ee5605f83d0bb0682be5bb1fcf141d12f615cac4ceb not found: ID does not exist" Oct 30 19:20:22.410855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.410817 6907 scope.go:122] "RemoveContainer" containerID="7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c" Oct 30 19:20:22.411085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411066 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c"} err="failed to get container status \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": rpc error: code = NotFound desc = could not find container \"7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c\": container with ID starting with 7e043d2b16c63710e7966deec5f053e7e21e6365b0819d0bdc20c6dea92ce54c not found: ID does not exist" Oct 30 19:20:22.411163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411087 6907 scope.go:122] "RemoveContainer" containerID="d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571" Oct 30 19:20:22.411325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411306 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571"} err="failed to get container status \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": rpc error: code = NotFound desc = could not find container \"d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571\": container with ID starting with d90a46a98d960f47a831387ddfdffe2955bfbc4d3054656c2da70a5eee300571 not found: ID does not exist" Oct 30 19:20:22.411372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411324 6907 scope.go:122] "RemoveContainer" containerID="58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6" Oct 30 19:20:22.411593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411573 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6"} err="failed to get container status \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": rpc error: code = NotFound desc = could not find container \"58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6\": container with ID starting with 58f5b8118c8a9089dbe2cc6ef2e417d89e7229d263c777ec0fb5880222a8d5b6 not found: ID does not exist" Oct 30 19:20:22.411670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411594 6907 scope.go:122] "RemoveContainer" containerID="82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e" Oct 30 19:20:22.411838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411821 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e"} err="failed to get container status \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": rpc error: code = NotFound desc = could not find container \"82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e\": container with ID starting with 82aa77446d61b99848b1c5a6b959051bd69e0363bc7e8274d70a5c166933a65e not found: ID does not exist" Oct 30 19:20:22.411877 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.411839 6907 scope.go:122] "RemoveContainer" containerID="0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc" Oct 30 19:20:22.412139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.412119 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc"} err="failed to get container status \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": rpc error: code = NotFound desc = could not find container \"0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc\": container with ID starting with 0c1acfa2071927c7302f434f61aa501dbc0fe3d6a1665792e3f0e2330b1c63bc not found: ID does not exist" Oct 30 19:20:22.412139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.412137 6907 scope.go:122] "RemoveContainer" containerID="d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953" Oct 30 19:20:22.412402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.412382 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953"} err="failed to get container status \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": rpc error: code = NotFound desc = could not find container \"d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953\": container with ID starting with d67aaa4c34811de915da88dc44ec83ee58cdcf545536ea24cb87b04a5a86d953 not found: ID does not exist" Oct 30 19:20:22.413168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413147 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:20:22.413168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413162 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:20:22.413168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413162 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:20:22.413351 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413179 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:20:22.413351 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413308 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413561 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413573 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413562 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413591 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8dg65acfiq0rj\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413606 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:20:22.413618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413607 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:20:22.413957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413612 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:20:22.413957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413613 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:20:22.413957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.413634 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gcr25\"" Oct 30 19:20:22.415581 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.415558 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:20:22.415676 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.415598 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:20:22.427264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.427241 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:22.486129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486094 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486131 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486157 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486228 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m978\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-kube-api-access-7m978\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486332 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486364 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486383 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486423 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486484 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486516 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486538 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486553 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486599 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486638 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486655 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486678 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486696 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486826 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486755 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486826 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486783 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.486891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.486853 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587264 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587317 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587358 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587376 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587402 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587430 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587474 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587509 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587540 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587564 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587584 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.587642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587630 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587663 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587688 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587716 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587737 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m978\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-kube-api-access-7m978\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587766 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587813 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587833 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.587867 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.588128 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.588595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.588570 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.589026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.588994 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.589129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.589045 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.589736 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.589707 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.590194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.590072 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.590596 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.590643 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591420 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591432 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591481 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591625 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591763 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/689f237d-9e75-4ba8-a205-c7bc295bf7a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.591955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.591930 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.592232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.592205 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.592903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.592877 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.593093 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.593074 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.593421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.593404 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/689f237d-9e75-4ba8-a205-c7bc295bf7a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.593543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.593528 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.599650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.599630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m978\" (UniqueName: \"kubernetes.io/projected/689f237d-9e75-4ba8-a205-c7bc295bf7a8-kube-api-access-7m978\") pod \"prometheus-k8s-0\" (UID: \"689f237d-9e75-4ba8-a205-c7bc295bf7a8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:22.728956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:22.728916 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:23.011530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:23.011489 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:20:23.015693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:20:23.015666 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689f237d_9e75_4ba8_a205_c7bc295bf7a8.slice/crio-79ccb98948ea4d6aca1e8c4664f15a261df291f94ee04d7fd407976392144666 WatchSource:0}: Error finding container 79ccb98948ea4d6aca1e8c4664f15a261df291f94ee04d7fd407976392144666: Status 404 returned error can't find the container with id 79ccb98948ea4d6aca1e8c4664f15a261df291f94ee04d7fd407976392144666 Oct 30 19:20:23.354070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:23.353986 6907 generic.go:356] "Generic (PLEG): container finished" podID="689f237d-9e75-4ba8-a205-c7bc295bf7a8" containerID="f81620f499e5eb3d300ee9505e8f0777762a035cfa0127616b436f1bcaffca84" exitCode=0 Oct 30 19:20:23.354070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:23.354042 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerDied","Data":"f81620f499e5eb3d300ee9505e8f0777762a035cfa0127616b436f1bcaffca84"} Oct 30 19:20:23.354264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:23.354090 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"79ccb98948ea4d6aca1e8c4664f15a261df291f94ee04d7fd407976392144666"} Oct 30 19:20:24.151566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.149639 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625638da-1fc7-46f4-86f6-f87b66e4cc1f" path="/var/lib/kubelet/pods/625638da-1fc7-46f4-86f6-f87b66e4cc1f/volumes" Oct 30 19:20:24.361631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361595 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"8c0ddd8567b73e87dd98d92e07ca7cb7a049299c3f9410e0aa895130d02e069a"} Oct 30 19:20:24.361782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361663 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"290ec543fdcd1747c7502218b985e19ef05f3b5549d5a65cc0cb8548a21478f3"} Oct 30 19:20:24.361782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361690 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"79eb258d5046f8a020dcccad3d654f40320bb7f4923283e119b051b23a733d76"} Oct 30 19:20:24.361782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361720 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"19199428ca36d3a54305c3954be892d73c385ad2341f2e71aef565650cb6241b"} Oct 30 19:20:24.361782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361744 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"d18a5a9740376dd476adb6d90e4ca29e33b55a38f35061f28105730427bc55d0"} Oct 30 19:20:24.361782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.361766 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"689f237d-9e75-4ba8-a205-c7bc295bf7a8","Type":"ContainerStarted","Data":"79d42e720a1ba98b416c8d1d9a561836a4c84771bfc40d0e80588e8af93f7727"} Oct 30 19:20:24.392091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:24.392030 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.375940792 podStartE2EDuration="2.392004285s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.063493ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:24.390775072 +0000 UTC m=+30.367001018" watchObservedRunningTime="2026-10-30 19:20:24.392004285 +0000 UTC m=+30.368230230" Oct 30 19:20:27.729511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:27.729462 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:29.327072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:29.327032 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:29.342828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:29.342801 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:29.396876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:20:29.396851 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:20:37.737774 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:20:37.737733 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:21:07.735772 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:21:07.735728 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:21:32.210667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.210634 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9"] Oct 30 19:21:32.215520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.215502 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.219810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.219785 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:21:32.219810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.219792 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:21:32.219810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.219792 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kstl2\"" Oct 30 19:21:32.225734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.225710 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9"] Oct 30 19:21:32.272792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.272751 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq7t\" (UniqueName: \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.272986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.272886 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.272986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.272963 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.314463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.314417 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx"] Oct 30 19:21:32.318015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.317999 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.326681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.326658 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-x46q9\"" Oct 30 19:21:32.331806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.331780 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx"] Oct 30 19:21:32.373574 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373540 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.373574 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373580 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.373815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373599 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.373815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373669 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.373920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373839 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq7t\" (UniqueName: \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.373920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.373862 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lq6r\" (UniqueName: \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.374028 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.374013 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.374104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.374080 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.383492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.383461 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq7t\" (UniqueName: \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.417627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.417591 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh"] Oct 30 19:21:32.421286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.421269 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.430605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.430585 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-p82mn\"" Oct 30 19:21:32.431953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.431914 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh"] Oct 30 19:21:32.474310 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474284 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsb4d\" (UniqueName: \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.474472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474321 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.474472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474460 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.474559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474532 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.474598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474567 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.474654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474637 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lq6r\" (UniqueName: \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.474881 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474862 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.474918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.474881 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.486837 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.486806 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lq6r\" (UniqueName: \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.511118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.511080 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth"] Oct 30 19:21:32.515034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.515014 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.518760 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.518734 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-6xbps\"" Oct 30 19:21:32.524077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.524056 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth"] Oct 30 19:21:32.532127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.532108 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:21:32.575548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575514 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.575738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575562 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.575738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575591 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncdhj\" (UniqueName: \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.575738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575653 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsb4d\" (UniqueName: \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.575913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575888 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.575964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575922 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.576025 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.575986 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.576187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.576172 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.594748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.594717 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsb4d\" (UniqueName: \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.634566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.634538 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:21:32.677491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.677173 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.677491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.677222 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.677491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.677247 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncdhj\" (UniqueName: \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.677795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.677665 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.677795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.677671 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.686891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.686853 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncdhj\" (UniqueName: \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.738840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.738762 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:21:32.812465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.812410 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9"] Oct 30 19:21:32.815694 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:21:32.815653 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae000aa_d333_4b29_a809_b98bc007189b.slice/crio-67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff WatchSource:0}: Error finding container 67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff: Status 404 returned error can't find the container with id 67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff Oct 30 19:21:32.817810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.817791 6907 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:21:32.832891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.832863 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:21:32.910040 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.910008 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx"] Oct 30 19:21:32.913216 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:21:32.913190 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60305c3e_b185_45ee_a655_a35603c07b1a.slice/crio-de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338 WatchSource:0}: Error finding container de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338: Status 404 returned error can't find the container with id de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338 Oct 30 19:21:32.945067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.945035 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw"] Oct 30 19:21:32.949640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.949621 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:32.953801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.953779 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-hbhtw\"" Oct 30 19:21:32.958015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.957951 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw"] Oct 30 19:21:32.980598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.980572 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwm7\" (UniqueName: \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:32.980734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.980637 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:32.980775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:32.980748 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.024059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.023975 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh"] Oct 30 19:21:33.026709 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:21:33.026681 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5669578a_f791_4363_b718_5339a4399bd9.slice/crio-82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a WatchSource:0}: Error finding container 82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a: Status 404 returned error can't find the container with id 82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a Oct 30 19:21:33.082137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.082099 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.082137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.082136 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwm7\" (UniqueName: \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.082398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.082214 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.082595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.082575 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.082665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.082591 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.093639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.093613 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwm7\" (UniqueName: \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.106709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.106679 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth"] Oct 30 19:21:33.110152 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:21:33.110126 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5106427c_9e9a_46e2_b66c_c25c311d627b.slice/crio-3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf WatchSource:0}: Error finding container 3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf: Status 404 returned error can't find the container with id 3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf Oct 30 19:21:33.268400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.268369 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:21:33.555988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.555945 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw"] Oct 30 19:21:33.626542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.626473 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" event={"ID":"fcbd3812-67c4-43dd-950c-89d22f1151e3","Type":"ContainerStarted","Data":"ce4dd160b07286cd67fdcafc0e93038231ca06194995f27248606fb322e0acb6"} Oct 30 19:21:33.628742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.628716 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" event={"ID":"60305c3e-b185-45ee-a655-a35603c07b1a","Type":"ContainerStarted","Data":"de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338"} Oct 30 19:21:33.630544 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.630485 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerStarted","Data":"67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff"} Oct 30 19:21:33.633568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.633541 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" event={"ID":"5669578a-f791-4363-b718-5339a4399bd9","Type":"ContainerStarted","Data":"82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a"} Oct 30 19:21:33.635537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:33.635506 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" event={"ID":"5106427c-9e9a-46e2-b66c-c25c311d627b","Type":"ContainerStarted","Data":"3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf"} Oct 30 19:21:34.820845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.818769 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt"] Oct 30 19:21:34.822982 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.822956 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:34.829947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.829906 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt"] Oct 30 19:21:34.832599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.832562 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-q2lbq\"" Oct 30 19:21:34.902401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.902353 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7srb\" (UniqueName: \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:34.902687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.902413 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:34.902687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:34.902468 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.003591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.003549 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7srb\" (UniqueName: \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.003591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.003601 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.003591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.003632 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.004063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.004045 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.004190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.004167 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.019215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.019178 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7srb\" (UniqueName: \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.146724 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.146388 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:21:35.444685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.444388 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt"] Oct 30 19:21:35.656387 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.656346 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" event={"ID":"bb9d0e93-5869-410d-9569-9c06f76614cb","Type":"ContainerStarted","Data":"b4e78b8cbdfc501c477acb9d0988607bf62866864d9d193b2496db4f0bccca7f"} Oct 30 19:21:35.862283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.862093 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng"] Oct 30 19:21:35.867397 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.867371 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.876121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.875973 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:21:35.876121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.876056 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:21:35.876329 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.876127 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:21:35.882523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.882495 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng"] Oct 30 19:21:35.897599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.897541 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbg4v\"" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912265 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfhm\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-kube-api-access-ptfhm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912314 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912365 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912402 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912431 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912486 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29f3ff5-991d-4ca7-af41-a592d6e767da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912512 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912554 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:35.912841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:35.912632 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014575 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29f3ff5-991d-4ca7-af41-a592d6e767da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014625 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014710 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014792 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014834 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfhm\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-kube-api-access-ptfhm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014860 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014916 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014953 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.014995 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.015697 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.016334 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d29f3ff5-991d-4ca7-af41-a592d6e767da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.017404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.016913 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.018850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.018750 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.019009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.018880 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.033341 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.033309 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.039582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.039495 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfhm\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-kube-api-access-ptfhm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.039832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.038869 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.042836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.042533 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d29f3ff5-991d-4ca7-af41-a592d6e767da-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-bcxng\" (UID: \"d29f3ff5-991d-4ca7-af41-a592d6e767da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.187998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.187912 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:36.505065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.505021 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng"] Oct 30 19:21:36.514598 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:21:36.514566 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29f3ff5_991d_4ca7_af41_a592d6e767da.slice/crio-785359ebd2d12d4e1ac148ae33aa346a811172c6d4189931e813b81f3c27adc8 WatchSource:0}: Error finding container 785359ebd2d12d4e1ac148ae33aa346a811172c6d4189931e813b81f3c27adc8: Status 404 returned error can't find the container with id 785359ebd2d12d4e1ac148ae33aa346a811172c6d4189931e813b81f3c27adc8 Oct 30 19:21:36.665749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:36.665705 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" event={"ID":"d29f3ff5-991d-4ca7-af41-a592d6e767da","Type":"ContainerStarted","Data":"785359ebd2d12d4e1ac148ae33aa346a811172c6d4189931e813b81f3c27adc8"} Oct 30 19:21:37.785033 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:21:37.784969 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:21:39.860328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.860292 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg"] Oct 30 19:21:39.865055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.864667 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:39.870921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.870887 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg"] Oct 30 19:21:39.965712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.965673 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5chs\" (UniqueName: \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:39.965914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.965729 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:39.965914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:39.965796 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.067292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.067245 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5chs\" (UniqueName: \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.067519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.067303 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.067519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.067356 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.067886 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.067856 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.068694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.068671 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.089091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.089017 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5chs\" (UniqueName: \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.191924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.191614 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:21:40.508373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.508335 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg"] Oct 30 19:21:40.719417 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:40.719373 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" event={"ID":"c70e0824-4370-4c66-9573-7eebb6d6be95","Type":"ContainerStarted","Data":"a32915488f28f878f0e3038a7f9b4062ad817bab56565db1e6846d2b4deab084"} Oct 30 19:21:43.673145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:43.673101 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:43.673647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:43.673228 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:43.673647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:43.673272 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:21:44.756565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:44.756528 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" event={"ID":"d29f3ff5-991d-4ca7-af41-a592d6e767da","Type":"ContainerStarted","Data":"6af1753faf6b2d9b77a8d2951dc6c60767ceac1c01c8b5f4b5033602867ea056"} Oct 30 19:21:44.777348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:44.776790 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podStartSLOduration=2.622091133 podStartE2EDuration="9.776771488s" totalImagesPullingTime="7.154680355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:44.775902116 +0000 UTC m=+110.752128060" watchObservedRunningTime="2026-10-30 19:21:44.776771488 +0000 UTC m=+110.752997437" Oct 30 19:21:45.188696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:45.188603 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:45.194328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:45.194295 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:45.781041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:45.780880 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:45.783291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:45.782974 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:21:46.789477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:46.788527 6907 generic.go:356] "Generic (PLEG): container finished" podID="8ae000aa-d333-4b29-a809-b98bc007189b" containerID="fada1692541f764fc6e27b0bbc9a8efac68125e4ef1adde69e3a673dae59987d" exitCode=0 Oct 30 19:21:46.789967 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:46.789697 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerDied","Data":"fada1692541f764fc6e27b0bbc9a8efac68125e4ef1adde69e3a673dae59987d"} Oct 30 19:21:47.795797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.795756 6907 generic.go:356] "Generic (PLEG): container finished" podID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerID="2ebaae3cd4550659bad29ddecb127175f964a558bde075bb464d17cca522012f" exitCode=0 Oct 30 19:21:47.796299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.795851 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" event={"ID":"fcbd3812-67c4-43dd-950c-89d22f1151e3","Type":"ContainerDied","Data":"2ebaae3cd4550659bad29ddecb127175f964a558bde075bb464d17cca522012f"} Oct 30 19:21:47.800240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.800086 6907 generic.go:356] "Generic (PLEG): container finished" podID="60305c3e-b185-45ee-a655-a35603c07b1a" containerID="643642a6f54a96230b4390391e842e75997bab3321061fcfeff7eb736aa825c5" exitCode=0 Oct 30 19:21:47.800240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.800189 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" event={"ID":"60305c3e-b185-45ee-a655-a35603c07b1a","Type":"ContainerDied","Data":"643642a6f54a96230b4390391e842e75997bab3321061fcfeff7eb736aa825c5"} Oct 30 19:21:47.802355 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.802331 6907 generic.go:356] "Generic (PLEG): container finished" podID="5669578a-f791-4363-b718-5339a4399bd9" containerID="9637cf6a9e26b1d917ef2abc6a4b5d846eba7e785be6a3d2017fd625e843a83e" exitCode=0 Oct 30 19:21:47.802550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.802502 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" event={"ID":"5669578a-f791-4363-b718-5339a4399bd9","Type":"ContainerDied","Data":"9637cf6a9e26b1d917ef2abc6a4b5d846eba7e785be6a3d2017fd625e843a83e"} Oct 30 19:21:47.805498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.804955 6907 generic.go:356] "Generic (PLEG): container finished" podID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerID="7d17fdaa3d71a165a21a8574f035e176103bd66ca5471f80dc65deec6abb547c" exitCode=0 Oct 30 19:21:47.805498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:47.804984 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" event={"ID":"5106427c-9e9a-46e2-b66c-c25c311d627b","Type":"ContainerDied","Data":"7d17fdaa3d71a165a21a8574f035e176103bd66ca5471f80dc65deec6abb547c"} Oct 30 19:21:48.811270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.811180 6907 generic.go:356] "Generic (PLEG): container finished" podID="60305c3e-b185-45ee-a655-a35603c07b1a" containerID="532ad80c7eb9fd532b1add07e9218ac4fcf975b49d3f1d3cf171d21364891414" exitCode=0 Oct 30 19:21:48.811270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.811217 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" event={"ID":"60305c3e-b185-45ee-a655-a35603c07b1a","Type":"ContainerDied","Data":"532ad80c7eb9fd532b1add07e9218ac4fcf975b49d3f1d3cf171d21364891414"} Oct 30 19:21:48.813257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.813173 6907 generic.go:356] "Generic (PLEG): container finished" podID="8ae000aa-d333-4b29-a809-b98bc007189b" containerID="3ee97c33b8b2f3f2f36db0e3cf1a388656f6f1976cd0bc2c28a690af41df7f7e" exitCode=0 Oct 30 19:21:48.813329 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.813251 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerDied","Data":"3ee97c33b8b2f3f2f36db0e3cf1a388656f6f1976cd0bc2c28a690af41df7f7e"} Oct 30 19:21:48.815191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.815077 6907 generic.go:356] "Generic (PLEG): container finished" podID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerID="b07302353d9e5289563305e60e236765c122034714082418ba59fa0eb01f3848" exitCode=0 Oct 30 19:21:48.815191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.815161 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" event={"ID":"bb9d0e93-5869-410d-9569-9c06f76614cb","Type":"ContainerDied","Data":"b07302353d9e5289563305e60e236765c122034714082418ba59fa0eb01f3848"} Oct 30 19:21:48.817079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.817061 6907 generic.go:356] "Generic (PLEG): container finished" podID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerID="b6e52f6b53d54d602a30305d4e9b24d558fc1c95b9aefc292545c50cd6ad90d0" exitCode=0 Oct 30 19:21:48.817163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.817142 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" event={"ID":"5106427c-9e9a-46e2-b66c-c25c311d627b","Type":"ContainerDied","Data":"b6e52f6b53d54d602a30305d4e9b24d558fc1c95b9aefc292545c50cd6ad90d0"} Oct 30 19:21:48.818740 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.818714 6907 generic.go:356] "Generic (PLEG): container finished" podID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerID="5a96141ea7aca9f1a658ae534c8ca3b4b009667e1ba2d5f2166590d14ee99c36" exitCode=0 Oct 30 19:21:48.818845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:48.818745 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" event={"ID":"c70e0824-4370-4c66-9573-7eebb6d6be95","Type":"ContainerDied","Data":"5a96141ea7aca9f1a658ae534c8ca3b4b009667e1ba2d5f2166590d14ee99c36"} Oct 30 19:21:49.829399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.829305 6907 generic.go:356] "Generic (PLEG): container finished" podID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerID="4a92c8a394ed1b3cf3b9e0c728384cb2bcd035a4a38f027bd6f4f4c09bb49b13" exitCode=0 Oct 30 19:21:49.829853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.829435 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" event={"ID":"c70e0824-4370-4c66-9573-7eebb6d6be95","Type":"ContainerDied","Data":"4a92c8a394ed1b3cf3b9e0c728384cb2bcd035a4a38f027bd6f4f4c09bb49b13"} Oct 30 19:21:49.833946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.833914 6907 generic.go:356] "Generic (PLEG): container finished" podID="5669578a-f791-4363-b718-5339a4399bd9" containerID="6ef030741a2580ad401acdc14b90a65a99904823391209dd1c6abd3c5bf04108" exitCode=0 Oct 30 19:21:49.834843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.834649 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" event={"ID":"5669578a-f791-4363-b718-5339a4399bd9","Type":"ContainerDied","Data":"6ef030741a2580ad401acdc14b90a65a99904823391209dd1c6abd3c5bf04108"} Oct 30 19:21:49.838972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.836877 6907 generic.go:356] "Generic (PLEG): container finished" podID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerID="019b143290be10c8ef95eaf7f2aaa411feb6d8f16eb9da71529c8b85b91367d4" exitCode=0 Oct 30 19:21:49.838972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:49.836955 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" event={"ID":"bb9d0e93-5869-410d-9569-9c06f76614cb","Type":"ContainerDied","Data":"019b143290be10c8ef95eaf7f2aaa411feb6d8f16eb9da71529c8b85b91367d4"} Oct 30 19:21:51.862125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:51.862081 6907 generic.go:356] "Generic (PLEG): container finished" podID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerID="9f2e5a9cb4c8382e6ff40955352aabd0154556ef1d654a2947b12a4fd585f13c" exitCode=0 Oct 30 19:21:51.862702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:21:51.862146 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" event={"ID":"fcbd3812-67c4-43dd-950c-89d22f1151e3","Type":"ContainerDied","Data":"9f2e5a9cb4c8382e6ff40955352aabd0154556ef1d654a2947b12a4fd585f13c"} Oct 30 19:22:03.981628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:03.981062 6907 generic.go:356] "Generic (PLEG): container finished" podID="60305c3e-b185-45ee-a655-a35603c07b1a" containerID="40ca271b1a15cbb19d7ce8f6410ac58c31f1484f4fad0c0dfc138573673315c5" exitCode=0 Oct 30 19:22:03.981628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:03.981162 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" event={"ID":"60305c3e-b185-45ee-a655-a35603c07b1a","Type":"ContainerDied","Data":"40ca271b1a15cbb19d7ce8f6410ac58c31f1484f4fad0c0dfc138573673315c5"} Oct 30 19:22:03.986014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:03.985413 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerStarted","Data":"f037f7d6ee656e1379ab181d4fba1f0c1ac6961f2b32f4297ebf5874dc1a202a"} Oct 30 19:22:04.022961 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.017565 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" podStartSLOduration=2.444704415 podStartE2EDuration="32.01754782s" totalImagesPullingTime="29.572843405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:04.017081962 +0000 UTC m=+129.993307906" watchObservedRunningTime="2026-10-30 19:22:04.01754782 +0000 UTC m=+129.993773764" Oct 30 19:22:04.405300 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.405269 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-92w59"] Oct 30 19:22:04.431177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.431136 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-92w59"] Oct 30 19:22:04.431340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.431268 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.434258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.434232 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/8098a0aa-023d-4ff8-903d-0568c404baef-kube-api-access-llgqj\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.434386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.434287 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8098a0aa-023d-4ff8-903d-0568c404baef-tmp\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.438290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.438266 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:22:04.438290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.438266 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:22:04.465279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.465236 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-9t9f2\"" Oct 30 19:22:04.535632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.535136 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8098a0aa-023d-4ff8-903d-0568c404baef-tmp\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.535632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.535282 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/8098a0aa-023d-4ff8-903d-0568c404baef-kube-api-access-llgqj\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.535632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.535520 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8098a0aa-023d-4ff8-903d-0568c404baef-tmp\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.545143 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.545123 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/8098a0aa-023d-4ff8-903d-0568c404baef-kube-api-access-llgqj\") pod \"jobset-operator-674cd59dc5-92w59\" (UID: \"8098a0aa-023d-4ff8-903d-0568c404baef\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.781399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.781370 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" Oct 30 19:22:04.992741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.992706 6907 generic.go:356] "Generic (PLEG): container finished" podID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerID="b1523d2f507d600e327ed4c2d50252f5517293ac67d15c58eb15ab729ca09e6a" exitCode=0 Oct 30 19:22:04.993188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.992794 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" event={"ID":"c70e0824-4370-4c66-9573-7eebb6d6be95","Type":"ContainerDied","Data":"b1523d2f507d600e327ed4c2d50252f5517293ac67d15c58eb15ab729ca09e6a"} Oct 30 19:22:04.995237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.995209 6907 generic.go:356] "Generic (PLEG): container finished" podID="8ae000aa-d333-4b29-a809-b98bc007189b" containerID="f037f7d6ee656e1379ab181d4fba1f0c1ac6961f2b32f4297ebf5874dc1a202a" exitCode=0 Oct 30 19:22:04.995400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.995303 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerDied","Data":"f037f7d6ee656e1379ab181d4fba1f0c1ac6961f2b32f4297ebf5874dc1a202a"} Oct 30 19:22:04.997083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.997061 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:22:04.997751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.997729 6907 generic.go:356] "Generic (PLEG): container finished" podID="5669578a-f791-4363-b718-5339a4399bd9" containerID="0e54b307e4da46c92e3c5917a0413d1cf8d8d2962512d1c746fac2fd2983f9f8" exitCode=1 Oct 30 19:22:04.997909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:04.997893 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" event={"ID":"5669578a-f791-4363-b718-5339a4399bd9","Type":"ContainerDied","Data":"0e54b307e4da46c92e3c5917a0413d1cf8d8d2962512d1c746fac2fd2983f9f8"} Oct 30 19:22:05.000180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.000159 6907 generic.go:356] "Generic (PLEG): container finished" podID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerID="282de9ebc4426ad0767e6c6939625fb9d04aa86a9a19b72714c056f88d512b91" exitCode=0 Oct 30 19:22:05.000318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.000220 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" event={"ID":"bb9d0e93-5869-410d-9569-9c06f76614cb","Type":"ContainerDied","Data":"282de9ebc4426ad0767e6c6939625fb9d04aa86a9a19b72714c056f88d512b91"} Oct 30 19:22:05.002625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.002601 6907 generic.go:356] "Generic (PLEG): container finished" podID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerID="644f61559df8a16b7dea572d0e41fcb01e78fd4e20a8c041b107eb0f5843ff06" exitCode=0 Oct 30 19:22:05.002731 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.002683 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" event={"ID":"5106427c-9e9a-46e2-b66c-c25c311d627b","Type":"ContainerDied","Data":"644f61559df8a16b7dea572d0e41fcb01e78fd4e20a8c041b107eb0f5843ff06"} Oct 30 19:22:05.065008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.064974 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-92w59"] Oct 30 19:22:05.075021 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:22:05.074988 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8098a0aa_023d_4ff8_903d_0568c404baef.slice/crio-4c13c6e16ed05c9bf69151dc35019c748f00a11f31af0018ef79a9bb8dbfda5d WatchSource:0}: Error finding container 4c13c6e16ed05c9bf69151dc35019c748f00a11f31af0018ef79a9bb8dbfda5d: Status 404 returned error can't find the container with id 4c13c6e16ed05c9bf69151dc35019c748f00a11f31af0018ef79a9bb8dbfda5d Oct 30 19:22:05.151596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.151565 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:22:05.241380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.241348 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\") pod \"60305c3e-b185-45ee-a655-a35603c07b1a\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " Oct 30 19:22:05.241568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.241418 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\" (UniqueName: \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\") pod \"60305c3e-b185-45ee-a655-a35603c07b1a\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " Oct 30 19:22:05.241568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.241506 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\") pod \"60305c3e-b185-45ee-a655-a35603c07b1a\" (UID: \"60305c3e-b185-45ee-a655-a35603c07b1a\") " Oct 30 19:22:05.241974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.241946 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle" pod "60305c3e-b185-45ee-a655-a35603c07b1a" (UID: "60305c3e-b185-45ee-a655-a35603c07b1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.244061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.244032 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r" pod "60305c3e-b185-45ee-a655-a35603c07b1a" (UID: "60305c3e-b185-45ee-a655-a35603c07b1a"). InnerVolumeSpecName "kube-api-access-5lq6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:05.247644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.247619 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util" pod "60305c3e-b185-45ee-a655-a35603c07b1a" (UID: "60305c3e-b185-45ee-a655-a35603c07b1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.342635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.342551 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.342635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.342584 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lq6r\" (UniqueName: \"kubernetes.io/projected/60305c3e-b185-45ee-a655-a35603c07b1a-kube-api-access-5lq6r\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.342635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:05.342600 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/60305c3e-b185-45ee-a655-a35603c07b1a-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.008934 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.008901 6907 generic.go:356] "Generic (PLEG): container finished" podID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerID="d95c4dbba77bd390341e80a619e1a9087c00e6d634f806aa1bbbff3d646b6083" exitCode=0 Oct 30 19:22:06.009350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.008996 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" event={"ID":"fcbd3812-67c4-43dd-950c-89d22f1151e3","Type":"ContainerDied","Data":"d95c4dbba77bd390341e80a619e1a9087c00e6d634f806aa1bbbff3d646b6083"} Oct 30 19:22:06.010863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.010839 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" event={"ID":"60305c3e-b185-45ee-a655-a35603c07b1a","Type":"ContainerDied","Data":"de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338"} Oct 30 19:22:06.010994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.010882 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7a570f8975cd836296d044213db8daf110dd00073dc1e38b2848b4f15fe338" Oct 30 19:22:06.010994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.010851 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76pwvx" Oct 30 19:22:06.012165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.012138 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" event={"ID":"8098a0aa-023d-4ff8-903d-0568c404baef","Type":"ContainerStarted","Data":"4c13c6e16ed05c9bf69151dc35019c748f00a11f31af0018ef79a9bb8dbfda5d"} Oct 30 19:22:06.223805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.223781 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:22:06.224923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.224657 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:22:06.226846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.226825 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:22:06.234005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.233982 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:22:06.259758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259725 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\" (UniqueName: \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\") pod \"5669578a-f791-4363-b718-5339a4399bd9\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " Oct 30 19:22:06.259927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259791 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\" (UniqueName: \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\") pod \"bb9d0e93-5869-410d-9569-9c06f76614cb\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " Oct 30 19:22:06.259927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259842 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\") pod \"8ae000aa-d333-4b29-a809-b98bc007189b\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " Oct 30 19:22:06.259927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259884 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\") pod \"5669578a-f791-4363-b718-5339a4399bd9\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " Oct 30 19:22:06.260095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259947 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\") pod \"8ae000aa-d333-4b29-a809-b98bc007189b\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " Oct 30 19:22:06.260095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.259984 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\") pod \"5669578a-f791-4363-b718-5339a4399bd9\" (UID: \"5669578a-f791-4363-b718-5339a4399bd9\") " Oct 30 19:22:06.260095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.260012 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\") pod \"bb9d0e93-5869-410d-9569-9c06f76614cb\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " Oct 30 19:22:06.260095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.260040 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\") pod \"bb9d0e93-5869-410d-9569-9c06f76614cb\" (UID: \"bb9d0e93-5869-410d-9569-9c06f76614cb\") " Oct 30 19:22:06.260095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.260081 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\" (UniqueName: \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\") pod \"8ae000aa-d333-4b29-a809-b98bc007189b\" (UID: \"8ae000aa-d333-4b29-a809-b98bc007189b\") " Oct 30 19:22:06.261193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.261165 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle" pod "5669578a-f791-4363-b718-5339a4399bd9" (UID: "5669578a-f791-4363-b718-5339a4399bd9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.266293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.266075 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t" pod "8ae000aa-d333-4b29-a809-b98bc007189b" (UID: "8ae000aa-d333-4b29-a809-b98bc007189b"). InnerVolumeSpecName "kube-api-access-dgq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:06.267301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.266980 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle" pod "bb9d0e93-5869-410d-9569-9c06f76614cb" (UID: "bb9d0e93-5869-410d-9569-9c06f76614cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.269066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.269032 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle" pod "8ae000aa-d333-4b29-a809-b98bc007189b" (UID: "8ae000aa-d333-4b29-a809-b98bc007189b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.269942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.269911 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util" pod "5669578a-f791-4363-b718-5339a4399bd9" (UID: "5669578a-f791-4363-b718-5339a4399bd9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.270090 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.270060 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d" pod "5669578a-f791-4363-b718-5339a4399bd9" (UID: "5669578a-f791-4363-b718-5339a4399bd9"). InnerVolumeSpecName "kube-api-access-rsb4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:06.272520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.272380 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util" pod "bb9d0e93-5869-410d-9569-9c06f76614cb" (UID: "bb9d0e93-5869-410d-9569-9c06f76614cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.273460 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.272601 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util" pod "8ae000aa-d333-4b29-a809-b98bc007189b" (UID: "8ae000aa-d333-4b29-a809-b98bc007189b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.274814 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.274765 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb" pod "bb9d0e93-5869-410d-9569-9c06f76614cb" (UID: "bb9d0e93-5869-410d-9569-9c06f76614cb"). InnerVolumeSpecName "kube-api-access-r7srb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:06.284948 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.284555 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:22:06.290158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.290067 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:22:06.361527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361497 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\") pod \"5106427c-9e9a-46e2-b66c-c25c311d627b\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " Oct 30 19:22:06.361705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361563 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\" (UniqueName: \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\") pod \"c70e0824-4370-4c66-9573-7eebb6d6be95\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " Oct 30 19:22:06.361705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361609 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\" (UniqueName: \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\") pod \"5106427c-9e9a-46e2-b66c-c25c311d627b\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " Oct 30 19:22:06.361705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361655 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\") pod \"5106427c-9e9a-46e2-b66c-c25c311d627b\" (UID: \"5106427c-9e9a-46e2-b66c-c25c311d627b\") " Oct 30 19:22:06.361705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361700 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\") pod \"c70e0824-4370-4c66-9573-7eebb6d6be95\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " Oct 30 19:22:06.361933 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.361739 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\") pod \"c70e0824-4370-4c66-9573-7eebb6d6be95\" (UID: \"c70e0824-4370-4c66-9573-7eebb6d6be95\") " Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362415 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsb4d\" (UniqueName: \"kubernetes.io/projected/5669578a-f791-4363-b718-5339a4399bd9-kube-api-access-rsb4d\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362472 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7srb\" (UniqueName: \"kubernetes.io/projected/bb9d0e93-5869-410d-9569-9c06f76614cb-kube-api-access-r7srb\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362490 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362501 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362509 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ae000aa-d333-4b29-a809-b98bc007189b-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362517 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5669578a-f791-4363-b718-5339a4399bd9-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362524 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362545 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb9d0e93-5869-410d-9569-9c06f76614cb-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362553 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgq7t\" (UniqueName: \"kubernetes.io/projected/8ae000aa-d333-4b29-a809-b98bc007189b-kube-api-access-dgq7t\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.362939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362796 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle" pod "c70e0824-4370-4c66-9573-7eebb6d6be95" (UID: "c70e0824-4370-4c66-9573-7eebb6d6be95"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.362939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.362865 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle" pod "5106427c-9e9a-46e2-b66c-c25c311d627b" (UID: "5106427c-9e9a-46e2-b66c-c25c311d627b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.364842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.364799 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs" pod "c70e0824-4370-4c66-9573-7eebb6d6be95" (UID: "c70e0824-4370-4c66-9573-7eebb6d6be95"). InnerVolumeSpecName "kube-api-access-n5chs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:06.366237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.366178 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj" pod "5106427c-9e9a-46e2-b66c-c25c311d627b" (UID: "5106427c-9e9a-46e2-b66c-c25c311d627b"). InnerVolumeSpecName "kube-api-access-ncdhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:06.368809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.368778 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util" pod "c70e0824-4370-4c66-9573-7eebb6d6be95" (UID: "c70e0824-4370-4c66-9573-7eebb6d6be95"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.370349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.370321 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util" pod "5106427c-9e9a-46e2-b66c-c25c311d627b" (UID: "5106427c-9e9a-46e2-b66c-c25c311d627b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:06.463330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463295 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncdhj\" (UniqueName: \"kubernetes.io/projected/5106427c-9e9a-46e2-b66c-c25c311d627b-kube-api-access-ncdhj\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.463330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463324 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.463330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463334 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.463330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463342 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70e0824-4370-4c66-9573-7eebb6d6be95-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.463697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463350 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5106427c-9e9a-46e2-b66c-c25c311d627b-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:06.463697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:06.463360 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5chs\" (UniqueName: \"kubernetes.io/projected/c70e0824-4370-4c66-9573-7eebb6d6be95-kube-api-access-n5chs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:07.019119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.019085 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" Oct 30 19:22:07.019608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.019084 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66ccvg" event={"ID":"c70e0824-4370-4c66-9573-7eebb6d6be95","Type":"ContainerDied","Data":"a32915488f28f878f0e3038a7f9b4062ad817bab56565db1e6846d2b4deab084"} Oct 30 19:22:07.019608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.019222 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32915488f28f878f0e3038a7f9b4062ad817bab56565db1e6846d2b4deab084" Oct 30 19:22:07.021419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.021387 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" event={"ID":"8ae000aa-d333-4b29-a809-b98bc007189b","Type":"ContainerDied","Data":"67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff"} Oct 30 19:22:07.021557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.021458 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67baea473da6b6c38a66c56ccc4093117b2bbdd7977dcbe4f5aa4a750237ffff" Oct 30 19:22:07.021557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.021408 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aggtp9" Oct 30 19:22:07.023117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.023057 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.023806 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.023825 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh" event={"ID":"5669578a-f791-4363-b718-5339a4399bd9","Type":"ContainerDied","Data":"82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a"} Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.023873 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ae09bc202fe49d7a6d9a40573b72aa6211729566702d28ac94bc70c7b1701a" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.025693 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.025708 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ltkbt" event={"ID":"bb9d0e93-5869-410d-9569-9c06f76614cb","Type":"ContainerDied","Data":"b4e78b8cbdfc501c477acb9d0988607bf62866864d9d193b2496db4f0bccca7f"} Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.025751 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e78b8cbdfc501c477acb9d0988607bf62866864d9d193b2496db4f0bccca7f" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.027986 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.028276 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdwnth" event={"ID":"5106427c-9e9a-46e2-b66c-c25c311d627b","Type":"ContainerDied","Data":"3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf"} Oct 30 19:22:07.028597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.028312 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ab0806222db2daba5a531e553d717fe369e3e6e5052456fa173078c2ff70bdf" Oct 30 19:22:07.230249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.229938 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:22:07.271334 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.271299 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\") pod \"fcbd3812-67c4-43dd-950c-89d22f1151e3\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " Oct 30 19:22:07.271565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.271369 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\") pod \"fcbd3812-67c4-43dd-950c-89d22f1151e3\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " Oct 30 19:22:07.271565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.271496 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\" (UniqueName: \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\") pod \"fcbd3812-67c4-43dd-950c-89d22f1151e3\" (UID: \"fcbd3812-67c4-43dd-950c-89d22f1151e3\") " Oct 30 19:22:07.272149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.272113 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle" pod "fcbd3812-67c4-43dd-950c-89d22f1151e3" (UID: "fcbd3812-67c4-43dd-950c-89d22f1151e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:07.273939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.273882 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7" pod "fcbd3812-67c4-43dd-950c-89d22f1151e3" (UID: "fcbd3812-67c4-43dd-950c-89d22f1151e3"). InnerVolumeSpecName "kube-api-access-2vwm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:07.277806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.277779 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util" pod "fcbd3812-67c4-43dd-950c-89d22f1151e3" (UID: "fcbd3812-67c4-43dd-950c-89d22f1151e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:07.372952 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.372901 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwm7\" (UniqueName: \"kubernetes.io/projected/fcbd3812-67c4-43dd-950c-89d22f1151e3-kube-api-access-2vwm7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:07.372952 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.372942 6907 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-bundle\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:07.372952 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:07.372958 6907 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcbd3812-67c4-43dd-950c-89d22f1151e3-util\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:07.738476 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:22:07.738370 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:22:08.043097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:08.043001 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" event={"ID":"fcbd3812-67c4-43dd-950c-89d22f1151e3","Type":"ContainerDied","Data":"ce4dd160b07286cd67fdcafc0e93038231ca06194995f27248606fb322e0acb6"} Oct 30 19:22:08.043097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:08.043055 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c8fvxw" Oct 30 19:22:08.043097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:08.043085 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce4dd160b07286cd67fdcafc0e93038231ca06194995f27248606fb322e0acb6" Oct 30 19:22:09.049556 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:09.049514 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" event={"ID":"8098a0aa-023d-4ff8-903d-0568c404baef","Type":"ContainerStarted","Data":"34b9df893a77b26a99ea847941c177bee411fd4efb6972b65313e55a7f9b4013"} Oct 30 19:22:09.070145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:09.070096 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-92w59" podStartSLOduration=1.9406198350000001 podStartE2EDuration="5.070081132s" totalImagesPullingTime="3.129461297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:09.069132289 +0000 UTC m=+135.045358268" watchObservedRunningTime="2026-10-30 19:22:09.070081132 +0000 UTC m=+135.046307075" Oct 30 19:22:12.084604 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084568 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b"] Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084885 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084895 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084906 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="extract" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084911 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="extract" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084920 6907 cpu_manager.go:414] "containerMap: removing container" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084925 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084941 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="pull" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084946 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="pull" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084956 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084960 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084966 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="pull" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084971 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="pull" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084976 6907 cpu_manager.go:414] "containerMap: removing container" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084982 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="util" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084988 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="extract" Oct 30 19:22:12.084991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.084993 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085001 6907 cpu_manager.go:414] "containerMap: removing container" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085005 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085011 6907 cpu_manager.go:414] "containerMap: removing container" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085016 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085022 6907 cpu_manager.go:414] "containerMap: removing container" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085026 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085032 6907 cpu_manager.go:414] "containerMap: removing container" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085036 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085042 6907 cpu_manager.go:414] "containerMap: removing container" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085046 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085053 6907 cpu_manager.go:414] "containerMap: removing container" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085057 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085065 6907 cpu_manager.go:414] "containerMap: removing container" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085069 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085075 6907 cpu_manager.go:414] "containerMap: removing container" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085079 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085089 6907 cpu_manager.go:414] "containerMap: removing container" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085094 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085100 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085104 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085109 6907 cpu_manager.go:414] "containerMap: removing container" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085113 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="util" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085119 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085123 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="pull" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085129 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085133 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085186 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5106427c-9e9a-46e2-b66c-c25c311d627b" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085193 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbd3812-67c4-43dd-950c-89d22f1151e3" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085199 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="60305c3e-b185-45ee-a655-a35603c07b1a" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085204 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9d0e93-5869-410d-9569-9c06f76614cb" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085210 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5669578a-f791-4363-b718-5339a4399bd9" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085217 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae000aa-d333-4b29-a809-b98bc007189b" containerName="extract" Oct 30 19:22:12.085487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.085224 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70e0824-4370-4c66-9573-7eebb6d6be95" containerName="extract" Oct 30 19:22:12.087375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.087359 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.095222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.095202 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zpjmc\"" Oct 30 19:22:12.099194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.099168 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:22:12.099311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.099216 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:22:12.099311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.099256 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:22:12.099739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.099719 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:22:12.099918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.099903 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:22:12.115743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115716 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfnk\" (UniqueName: \"kubernetes.io/projected/0032836d-a143-44b5-a034-249e94c41a76-kube-api-access-wlfnk\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.115873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115754 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-tmp\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.115873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115775 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-metrics-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.115958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115883 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.115958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115903 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-webhook-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.115958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.115942 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-manifests\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.121225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.121203 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b"] Oct 30 19:22:12.217363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217322 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfnk\" (UniqueName: \"kubernetes.io/projected/0032836d-a143-44b5-a034-249e94c41a76-kube-api-access-wlfnk\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.217596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217486 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-tmp\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.217596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217530 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-metrics-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.217596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217574 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.217596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217589 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-webhook-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.217821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.217662 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-manifests\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.218071 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.218048 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-manifests\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.219946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.219922 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0032836d-a143-44b5-a034-249e94c41a76-tmp\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.220189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.220163 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-metrics-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.220263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.220196 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-webhook-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.220263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.220243 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0032836d-a143-44b5-a034-249e94c41a76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.226624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.226600 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfnk\" (UniqueName: \"kubernetes.io/projected/0032836d-a143-44b5-a034-249e94c41a76-kube-api-access-wlfnk\") pod \"opendatahub-operator-controller-manager-668f68f55d-mzw7b\" (UID: \"0032836d-a143-44b5-a034-249e94c41a76\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.405801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.405699 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:12.775550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:12.775512 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b"] Oct 30 19:22:13.069135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:13.069048 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" event={"ID":"0032836d-a143-44b5-a034-249e94c41a76","Type":"ContainerStarted","Data":"6961ba561c50cb7180336bf9c57d2a9e6b21564b30301fbea11eb4c9421e0a10"} Oct 30 19:22:16.085350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:16.085303 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" event={"ID":"0032836d-a143-44b5-a034-249e94c41a76","Type":"ContainerStarted","Data":"c21cb60534fe1dfe66f8f1b51efa4f9d5f6049a519bf5a34bbb1c2aaa5691963"} Oct 30 19:22:17.092012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:17.091981 6907 generic.go:356] "Generic (PLEG): container finished" podID="0032836d-a143-44b5-a034-249e94c41a76" containerID="c21cb60534fe1dfe66f8f1b51efa4f9d5f6049a519bf5a34bbb1c2aaa5691963" exitCode=0 Oct 30 19:22:17.092422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:17.092057 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" event={"ID":"0032836d-a143-44b5-a034-249e94c41a76","Type":"ContainerDied","Data":"c21cb60534fe1dfe66f8f1b51efa4f9d5f6049a519bf5a34bbb1c2aaa5691963"} Oct 30 19:22:18.098646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:18.098607 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" event={"ID":"0032836d-a143-44b5-a034-249e94c41a76","Type":"ContainerStarted","Data":"b4a9349ea9fe91eb7f63ff411bc5b42611d6fe929070ffb0364c21dfd799d6bb"} Oct 30 19:22:18.099040 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:18.098836 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:18.160710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:18.160656 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" podStartSLOduration=2.292554675 podStartE2EDuration="6.160642383s" totalImagesPullingTime="3.302347974s" totalInitContainerRuntime="565.739734ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:18.157593789 +0000 UTC m=+144.133819735" watchObservedRunningTime="2026-10-30 19:22:18.160642383 +0000 UTC m=+144.136868327" Oct 30 19:22:25.658763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.658725 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4"] Oct 30 19:22:25.665204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.665184 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.676007 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.675980 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:22:25.676157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.676140 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-65gk2\"" Oct 30 19:22:25.676209 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.676194 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:22:25.677170 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.677147 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4"] Oct 30 19:22:25.746935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.746894 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7s6\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-kube-api-access-8v7s6\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.747121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.746973 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.847858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.847809 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.848029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.847897 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7s6\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-kube-api-access-8v7s6\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.858570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.858540 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.860685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.860660 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7s6\" (UniqueName: \"kubernetes.io/projected/cfb8121c-04df-44f7-a66e-59df415cf0f7-kube-api-access-8v7s6\") pod \"cert-manager-webhook-5dbbbf75cc-k2kb4\" (UID: \"cfb8121c-04df-44f7-a66e-59df415cf0f7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:25.976891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:25.976861 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:26.253489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:26.253394 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4"] Oct 30 19:22:26.256542 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:22:26.256512 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb8121c_04df_44f7_a66e_59df415cf0f7.slice/crio-836cdfdb66db43ea6c3afdb870d57b00b69c5d5160c970085963b80ceb07376b WatchSource:0}: Error finding container 836cdfdb66db43ea6c3afdb870d57b00b69c5d5160c970085963b80ceb07376b: Status 404 returned error can't find the container with id 836cdfdb66db43ea6c3afdb870d57b00b69c5d5160c970085963b80ceb07376b Oct 30 19:22:27.146738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:27.146704 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" event={"ID":"cfb8121c-04df-44f7-a66e-59df415cf0f7","Type":"ContainerStarted","Data":"836cdfdb66db43ea6c3afdb870d57b00b69c5d5160c970085963b80ceb07376b"} Oct 30 19:22:29.106415 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:29.106384 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-668f68f55d-mzw7b" Oct 30 19:22:30.164579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:30.164542 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" event={"ID":"cfb8121c-04df-44f7-a66e-59df415cf0f7","Type":"ContainerStarted","Data":"a039b52e6cfa1fa507df11b3f69ac9f3fed61023567cce39fc3c047ff8497974"} Oct 30 19:22:30.165031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:30.164635 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:30.186995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:30.186946 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" podStartSLOduration=2.139160244 podStartE2EDuration="5.186932017s" totalImagesPullingTime="3.047771773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:30.185638733 +0000 UTC m=+156.161864669" watchObservedRunningTime="2026-10-30 19:22:30.186932017 +0000 UTC m=+156.163157962" Oct 30 19:22:36.175689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.175657 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" Oct 30 19:22:36.191753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.191719 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282"] Oct 30 19:22:36.194883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.194859 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:36.204320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.204294 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gn75h\"" Oct 30 19:22:36.204482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.204431 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:22:36.204604 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.204552 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:22:36.214429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.214399 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282"] Oct 30 19:22:36.256579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.256546 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm26j\" (UniqueName: \"kubernetes.io/projected/f691bc00-6099-461b-a432-61ee33696e55-kube-api-access-fm26j\") pod \"limitador-operator-controller-manager-5778bb4449-b8282\" (UID: \"f691bc00-6099-461b-a432-61ee33696e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:36.357759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.357715 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm26j\" (UniqueName: \"kubernetes.io/projected/f691bc00-6099-461b-a432-61ee33696e55-kube-api-access-fm26j\") pod \"limitador-operator-controller-manager-5778bb4449-b8282\" (UID: \"f691bc00-6099-461b-a432-61ee33696e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:36.371095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.371059 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm26j\" (UniqueName: \"kubernetes.io/projected/f691bc00-6099-461b-a432-61ee33696e55-kube-api-access-fm26j\") pod \"limitador-operator-controller-manager-5778bb4449-b8282\" (UID: \"f691bc00-6099-461b-a432-61ee33696e55\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:36.507832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.507789 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:36.784937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:36.784792 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282"] Oct 30 19:22:36.787921 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:22:36.787890 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf691bc00_6099_461b_a432_61ee33696e55.slice/crio-d2d96989c09f936420d8767448d77c9ca1a5e43eab998a5b5acdd5d291b9735c WatchSource:0}: Error finding container d2d96989c09f936420d8767448d77c9ca1a5e43eab998a5b5acdd5d291b9735c: Status 404 returned error can't find the container with id d2d96989c09f936420d8767448d77c9ca1a5e43eab998a5b5acdd5d291b9735c Oct 30 19:22:37.212741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.212650 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" event={"ID":"f691bc00-6099-461b-a432-61ee33696e55","Type":"ContainerStarted","Data":"d2d96989c09f936420d8767448d77c9ca1a5e43eab998a5b5acdd5d291b9735c"} Oct 30 19:22:37.640371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.640337 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tppdh"] Oct 30 19:22:37.674006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.673971 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tppdh"] Oct 30 19:22:37.674176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.674070 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.734549 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:22:37.734505 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:22:37.765134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.765103 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-rrzgm\"" Oct 30 19:22:37.767742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.767719 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-bound-sa-token\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.767877 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.767768 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwgc\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-kube-api-access-kdwgc\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.869031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.868990 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-bound-sa-token\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.869205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.869071 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwgc\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-kube-api-access-kdwgc\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.878316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.878284 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-bound-sa-token\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.878474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.878358 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwgc\" (UniqueName: \"kubernetes.io/projected/582d9da6-2d46-4735-9649-febe6ea98937-kube-api-access-kdwgc\") pod \"cert-manager-597b95cb84-tppdh\" (UID: \"582d9da6-2d46-4735-9649-febe6ea98937\") " pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:37.999199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:37.999160 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tppdh" Oct 30 19:22:38.327846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:38.327808 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tppdh"] Oct 30 19:22:38.330872 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:22:38.330833 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod582d9da6_2d46_4735_9649_febe6ea98937.slice/crio-2067fba5d37f720a9690c2f698e82809ed040103011666e0842cbb96b1cf8329 WatchSource:0}: Error finding container 2067fba5d37f720a9690c2f698e82809ed040103011666e0842cbb96b1cf8329: Status 404 returned error can't find the container with id 2067fba5d37f720a9690c2f698e82809ed040103011666e0842cbb96b1cf8329 Oct 30 19:22:39.228528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:39.228482 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tppdh" event={"ID":"582d9da6-2d46-4735-9649-febe6ea98937","Type":"ContainerStarted","Data":"825e387e104b9b98c2c5cdb934671bec9a018f3c61862b50511f53b24c3c09ad"} Oct 30 19:22:39.228717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:39.228541 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tppdh" event={"ID":"582d9da6-2d46-4735-9649-febe6ea98937","Type":"ContainerStarted","Data":"2067fba5d37f720a9690c2f698e82809ed040103011666e0842cbb96b1cf8329"} Oct 30 19:22:39.249898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:39.249847 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tppdh" podStartSLOduration=2.249832136 podStartE2EDuration="2.249832136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:39.249419472 +0000 UTC m=+165.225645417" watchObservedRunningTime="2026-10-30 19:22:39.249832136 +0000 UTC m=+165.226058079" Oct 30 19:22:40.236341 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:40.236286 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" event={"ID":"f691bc00-6099-461b-a432-61ee33696e55","Type":"ContainerStarted","Data":"57de34820027a9698ab1960d16d75845bc1294d622e1c9f2229b5c55d839bb8e"} Oct 30 19:22:40.236860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:40.236370 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:40.282510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:40.282424 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" podStartSLOduration=1.131840102 podStartE2EDuration="4.282402527s" totalImagesPullingTime="3.150562425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:40.280494996 +0000 UTC m=+166.256720931" watchObservedRunningTime="2026-10-30 19:22:40.282402527 +0000 UTC m=+166.258628472" Oct 30 19:22:46.961938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:46.961901 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6"] Oct 30 19:22:46.966982 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:46.966958 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:46.975331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:46.974902 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6"] Oct 30 19:22:47.021034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.020999 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:22:47.021239 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.021138 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:22:47.021239 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.021222 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-qrxwc\"" Oct 30 19:22:47.055820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.055777 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88351064-cc56-4128-a7af-708e83790454-tmp\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.055993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.055839 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps42l\" (UniqueName: \"kubernetes.io/projected/88351064-cc56-4128-a7af-708e83790454-kube-api-access-ps42l\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.055993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.055943 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88351064-cc56-4128-a7af-708e83790454-tls-certs\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.156415 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.156367 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88351064-cc56-4128-a7af-708e83790454-tmp\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.156637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.156425 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps42l\" (UniqueName: \"kubernetes.io/projected/88351064-cc56-4128-a7af-708e83790454-kube-api-access-ps42l\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.156637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.156485 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88351064-cc56-4128-a7af-708e83790454-tls-certs\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.158656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.158629 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88351064-cc56-4128-a7af-708e83790454-tmp\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.158903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.158885 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88351064-cc56-4128-a7af-708e83790454-tls-certs\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.177605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.177569 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps42l\" (UniqueName: \"kubernetes.io/projected/88351064-cc56-4128-a7af-708e83790454-kube-api-access-ps42l\") pod \"kube-auth-proxy-6944dd5f6b-cz7g6\" (UID: \"88351064-cc56-4128-a7af-708e83790454\") " pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.280250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.280220 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" Oct 30 19:22:47.580003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:47.579835 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6"] Oct 30 19:22:48.275094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:48.275046 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" event={"ID":"88351064-cc56-4128-a7af-708e83790454","Type":"ContainerStarted","Data":"1e2aa4463143d3de71e4e68d39cbf98618e5a86874ba7cb385276effd7258e86"} Oct 30 19:22:51.243189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:51.243156 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-b8282" Oct 30 19:22:51.292044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:51.292004 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" event={"ID":"88351064-cc56-4128-a7af-708e83790454","Type":"ContainerStarted","Data":"af22e3b6259f4efa6197f59535afc100d5df4cb325b0fd3b6baa24d0b6ad0950"} Oct 30 19:22:51.309469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:22:51.309404 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6944dd5f6b-cz7g6" podStartSLOduration=2.675584703 podStartE2EDuration="5.309388719s" totalImagesPullingTime="2.633804016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:22:51.308146126 +0000 UTC m=+177.284372071" watchObservedRunningTime="2026-10-30 19:22:51.309388719 +0000 UTC m=+177.285614656" Oct 30 19:23:02.213814 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.213777 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:23:02.217314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.217291 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.221628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.221595 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5ptsd\"" Oct 30 19:23:02.226594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.226567 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:23:02.242686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.242665 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:23:02.268527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.268495 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:23:02.299781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.299746 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvsm\" (UniqueName: \"kubernetes.io/projected/afc2a5cc-51af-4df2-ba60-5dae4e376238-kube-api-access-cbvsm\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.299947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.299800 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afc2a5cc-51af-4df2-ba60-5dae4e376238-config-file\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.400270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.400220 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvsm\" (UniqueName: \"kubernetes.io/projected/afc2a5cc-51af-4df2-ba60-5dae4e376238-kube-api-access-cbvsm\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.400486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.400286 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afc2a5cc-51af-4df2-ba60-5dae4e376238-config-file\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.400972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.400951 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afc2a5cc-51af-4df2-ba60-5dae4e376238-config-file\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.409812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.409781 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvsm\" (UniqueName: \"kubernetes.io/projected/afc2a5cc-51af-4df2-ba60-5dae4e376238-kube-api-access-cbvsm\") pod \"limitador-limitador-7c9b87d574-749qf\" (UID: \"afc2a5cc-51af-4df2-ba60-5dae4e376238\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.531835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.531791 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:02.814592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:02.814494 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:23:02.816903 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:23:02.816863 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc2a5cc_51af_4df2_ba60_5dae4e376238.slice/crio-d5c8e9b3deca19850cd710fc1ed1b1da589b764867e29c377ae6565659205f42 WatchSource:0}: Error finding container d5c8e9b3deca19850cd710fc1ed1b1da589b764867e29c377ae6565659205f42: Status 404 returned error can't find the container with id d5c8e9b3deca19850cd710fc1ed1b1da589b764867e29c377ae6565659205f42 Oct 30 19:23:03.349421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:03.349383 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" event={"ID":"afc2a5cc-51af-4df2-ba60-5dae4e376238","Type":"ContainerStarted","Data":"d5c8e9b3deca19850cd710fc1ed1b1da589b764867e29c377ae6565659205f42"} Oct 30 19:23:04.356044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:04.356002 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" event={"ID":"afc2a5cc-51af-4df2-ba60-5dae4e376238","Type":"ContainerStarted","Data":"07ec4a55c1a55d020720d54cc0f212112d872df0344c8b889d36a17df95eefb3"} Oct 30 19:23:04.356519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:04.356126 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:07.734872 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:23:07.734830 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:23:15.362353 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:15.362315 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" Oct 30 19:23:15.379625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:15.379564 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-749qf" podStartSLOduration=12.012849762 podStartE2EDuration="13.379549331s" totalImagesPullingTime="1.366699569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:04.375591582 +0000 UTC m=+190.351817526" watchObservedRunningTime="2026-10-30 19:23:15.379549331 +0000 UTC m=+201.355775275" Oct 30 19:23:37.735300 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:23:37.735262 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:23:42.608648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.608607 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wfznm"] Oct 30 19:23:42.612865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.612844 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.623314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.623292 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:23:42.623424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.623350 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:23:42.626529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.626502 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wfznm"] Oct 30 19:23:42.650038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.650005 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-hnbrx"] Oct 30 19:23:42.653723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.653705 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.666082 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.666054 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hnbrx"] Oct 30 19:23:42.670907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.670879 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhrx\" (UniqueName: \"kubernetes.io/projected/77cf09e2-a980-402f-9685-9727db0b70ee-kube-api-access-plhrx\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.671035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.670925 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cf09e2-a980-402f-9685-9727db0b70ee-tls-certs\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.708237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.708208 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 19:23:42.708405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.708243 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 19:23:42.708405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.708284 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cj9fs\"" Oct 30 19:23:42.708405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.708308 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-tv7z5\"" Oct 30 19:23:42.708713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.708697 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 19:23:42.772187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.772144 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6r5\" (UniqueName: \"kubernetes.io/projected/ec4c3328-a427-4baa-9205-6cd30fcc5770-kube-api-access-kw6r5\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.772360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.772199 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhrx\" (UniqueName: \"kubernetes.io/projected/77cf09e2-a980-402f-9685-9727db0b70ee-kube-api-access-plhrx\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.772360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.772219 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-metrics-certs\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.772360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.772337 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cf09e2-a980-402f-9685-9727db0b70ee-tls-certs\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.772480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.772385 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-cert\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.774725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.774699 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cf09e2-a980-402f-9685-9727db0b70ee-tls-certs\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.783624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.783601 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhrx\" (UniqueName: \"kubernetes.io/projected/77cf09e2-a980-402f-9685-9727db0b70ee-kube-api-access-plhrx\") pod \"model-serving-api-75bd89fdfc-wfznm\" (UID: \"77cf09e2-a980-402f-9685-9727db0b70ee\") " pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.873911 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.873812 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-cert\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.874094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.873986 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6r5\" (UniqueName: \"kubernetes.io/projected/ec4c3328-a427-4baa-9205-6cd30fcc5770-kube-api-access-kw6r5\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.874094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.874019 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-metrics-certs\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.876403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.876378 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-cert\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.876560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.876405 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4c3328-a427-4baa-9205-6cd30fcc5770-metrics-certs\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.883605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.883574 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6r5\" (UniqueName: \"kubernetes.io/projected/ec4c3328-a427-4baa-9205-6cd30fcc5770-kube-api-access-kw6r5\") pod \"odh-model-controller-769875db56-hnbrx\" (UID: \"ec4c3328-a427-4baa-9205-6cd30fcc5770\") " pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:42.926634 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.926593 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:42.968164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:42.968126 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:43.211098 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.211068 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wfznm"] Oct 30 19:23:43.214002 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:23:43.213968 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cf09e2_a980_402f_9685_9727db0b70ee.slice/crio-345c7eec893ec74536e5df1a681374cbc70a7818b18fc4a265ce76702001685e WatchSource:0}: Error finding container 345c7eec893ec74536e5df1a681374cbc70a7818b18fc4a265ce76702001685e: Status 404 returned error can't find the container with id 345c7eec893ec74536e5df1a681374cbc70a7818b18fc4a265ce76702001685e Oct 30 19:23:43.263664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.261771 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-hnbrx"] Oct 30 19:23:43.545155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.545122 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2"] Oct 30 19:23:43.550368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.550341 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.553402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.553372 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" event={"ID":"77cf09e2-a980-402f-9685-9727db0b70ee","Type":"ContainerStarted","Data":"345c7eec893ec74536e5df1a681374cbc70a7818b18fc4a265ce76702001685e"} Oct 30 19:23:43.554512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.554479 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hnbrx" event={"ID":"ec4c3328-a427-4baa-9205-6cd30fcc5770","Type":"ContainerStarted","Data":"2486a875e85f5e5c08950df8b389c2b18cc86c82b9f0303811bd84dfc2aa7ab9"} Oct 30 19:23:43.562177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.560473 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:23:43.562177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.560509 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:23:43.562177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.560522 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:23:43.562177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.561983 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2"] Oct 30 19:23:43.562177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.562116 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-t7289\"" Oct 30 19:23:43.684071 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684026 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c0004a-8bf5-4cb7-9d05-411d92b85476-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684071 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684151 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684300 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684350 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684411 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p55z\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-kube-api-access-5p55z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.684571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.684475 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785582 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785644 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785674 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p55z\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-kube-api-access-5p55z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785703 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785757 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c0004a-8bf5-4cb7-9d05-411d92b85476-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785785 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.785953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.785828 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.787629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.787594 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.789625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.789595 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.789737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.789630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.789808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.789738 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.789983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.789961 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/75c0004a-8bf5-4cb7-9d05-411d92b85476-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.795522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.795421 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.796068 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.796042 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p55z\" (UniqueName: \"kubernetes.io/projected/75c0004a-8bf5-4cb7-9d05-411d92b85476-kube-api-access-5p55z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ld9n2\" (UID: \"75c0004a-8bf5-4cb7-9d05-411d92b85476\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:43.864874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:43.864799 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:44.200110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:44.190759 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2"] Oct 30 19:23:44.200110 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:23:44.193703 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c0004a_8bf5_4cb7_9d05_411d92b85476.slice/crio-5c448b90d4b3d89940a0daa3a5b887fcf5ada42961785f54cc15c4b5b75ea041 WatchSource:0}: Error finding container 5c448b90d4b3d89940a0daa3a5b887fcf5ada42961785f54cc15c4b5b75ea041: Status 404 returned error can't find the container with id 5c448b90d4b3d89940a0daa3a5b887fcf5ada42961785f54cc15c4b5b75ea041 Oct 30 19:23:44.569962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:44.565746 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" event={"ID":"75c0004a-8bf5-4cb7-9d05-411d92b85476","Type":"ContainerStarted","Data":"5c448b90d4b3d89940a0daa3a5b887fcf5ada42961785f54cc15c4b5b75ea041"} Oct 30 19:23:45.577909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:45.577865 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" event={"ID":"77cf09e2-a980-402f-9685-9727db0b70ee","Type":"ContainerStarted","Data":"cff7bc18c2ea9884d3d29392c33014cd9f2d6d587e8bf621e606248177f8c0b7"} Oct 30 19:23:45.578985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:45.578956 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:23:45.598153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:45.598090 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" podStartSLOduration=1.447603099 podStartE2EDuration="3.598070963s" totalImagesPullingTime="2.150467864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:45.597489009 +0000 UTC m=+231.573714953" watchObservedRunningTime="2026-10-30 19:23:45.598070963 +0000 UTC m=+231.574296911" Oct 30 19:23:46.586984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:46.586931 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-hnbrx" event={"ID":"ec4c3328-a427-4baa-9205-6cd30fcc5770","Type":"ContainerStarted","Data":"131d791b3e874db722d2d3829d2af6757a3c9c7dd37942bff32de7866d453fac"} Oct 30 19:23:46.587460 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:46.587021 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:46.624081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:46.624026 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-hnbrx" podStartSLOduration=1.470815072 podStartE2EDuration="4.624009235s" totalImagesPullingTime="3.153194163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:46.621254881 +0000 UTC m=+232.597480827" watchObservedRunningTime="2026-10-30 19:23:46.624009235 +0000 UTC m=+232.600235235" Oct 30 19:23:47.333315 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:47.333285 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:23:47.333400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:47.333351 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:23:47.594271 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:47.594169 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" event={"ID":"75c0004a-8bf5-4cb7-9d05-411d92b85476","Type":"ContainerStarted","Data":"0b8872c02f8d5abee98f77f167bda92658cbdc2f02611f08796f9e2da2a34339"} Oct 30 19:23:47.594713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:47.594634 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:47.626570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:47.626473 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" podStartSLOduration=1.491001089 podStartE2EDuration="4.626430904s" totalImagesPullingTime="3.135429815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:47.624072616 +0000 UTC m=+233.600298563" watchObservedRunningTime="2026-10-30 19:23:47.626430904 +0000 UTC m=+233.602656848" Oct 30 19:23:48.600773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:48.600744 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ld9n2" Oct 30 19:23:57.597104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:57.597073 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-hnbrx" Oct 30 19:23:57.598934 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:23:57.598910 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-wfznm" Oct 30 19:24:07.735185 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:24:07.735144 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:24:18.336235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:18.336195 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:24:28.807656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.807619 6907 generic.go:356] "Generic (PLEG): container finished" podID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerID="6af1753faf6b2d9b77a8d2951dc6c60767ceac1c01c8b5f4b5033602867ea056" exitCode=0 Oct 30 19:24:28.808091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.807673 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" event={"ID":"d29f3ff5-991d-4ca7-af41-a592d6e767da","Type":"ContainerDied","Data":"6af1753faf6b2d9b77a8d2951dc6c60767ceac1c01c8b5f4b5033602867ea056"} Oct 30 19:24:28.808091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.808032 6907 scope.go:122] "RemoveContainer" containerID="6af1753faf6b2d9b77a8d2951dc6c60767ceac1c01c8b5f4b5033602867ea056" Oct 30 19:24:28.809190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.809159 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:24:28.809256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.809237 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:24:28.809293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:28.809271 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:24:29.814686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:29.814653 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" event={"ID":"d29f3ff5-991d-4ca7-af41-a592d6e767da","Type":"ContainerStarted","Data":"754f94179d26a9c86c8cb6c8865d509dad05f27142be4bc5c86c73531e5e4076"} Oct 30 19:24:30.188740 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:30.188642 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:24:30.189856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:30.189830 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Oct 30 19:24:30.189923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:30.189892 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Oct 30 19:24:30.781565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:30.781524 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:24:31.188679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:31.188589 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Oct 30 19:24:31.188679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:31.188657 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Oct 30 19:24:32.188691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:32.188654 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Oct 30 19:24:32.189095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:32.188721 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Oct 30 19:24:33.189386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:33.189336 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Oct 30 19:24:33.189825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:33.189415 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Oct 30 19:24:34.188864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:34.188830 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body= Oct 30 19:24:34.189045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:34.188882 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" Oct 30 19:24:35.192723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:35.192691 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:24:35.193615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:35.193598 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" Oct 30 19:24:37.735873 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:24:37.735776 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:24:54.175062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:54.175028 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:24:54.176160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:54.176142 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:24:54.195219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:54.195195 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:24:54.196307 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:24:54.196285 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:25:07.735769 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:25:07.735732 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:25:37.736572 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:25:37.736534 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:25:45.445023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:45.444977 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:25:47.883368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:47.883328 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:25:49.729812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:49.729782 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:25:52.621586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:52.621542 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:25:57.056822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.056784 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:25:57.104210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.104177 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w"] Oct 30 19:25:57.108132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.108108 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.117486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.117455 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:25:57.117920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.117900 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:25:57.118004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.117945 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 19:25:57.118047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.118028 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ctlhc\"" Oct 30 19:25:57.123752 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.123727 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w"] Oct 30 19:25:57.159081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159045 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52a7fed-385d-45b9-a1a4-78be7a912e60-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.159263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159100 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.159339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159265 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpgl\" (UniqueName: \"kubernetes.io/projected/c52a7fed-385d-45b9-a1a4-78be7a912e60-kube-api-access-xkpgl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.159339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159318 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.159488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159461 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.159557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.159502 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263251 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpgl\" (UniqueName: \"kubernetes.io/projected/c52a7fed-385d-45b9-a1a4-78be7a912e60-kube-api-access-xkpgl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263318 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263360 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263381 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263459 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52a7fed-385d-45b9-a1a4-78be7a912e60-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263509 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.263993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.263960 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.264128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.264061 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.264234 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.264212 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.266100 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.266076 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c52a7fed-385d-45b9-a1a4-78be7a912e60-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.266301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.266278 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52a7fed-385d-45b9-a1a4-78be7a912e60-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.272628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.272603 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpgl\" (UniqueName: \"kubernetes.io/projected/c52a7fed-385d-45b9-a1a4-78be7a912e60-kube-api-access-xkpgl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w\" (UID: \"c52a7fed-385d-45b9-a1a4-78be7a912e60\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.422258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.422167 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:25:57.709783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:57.709750 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w"] Oct 30 19:25:57.711931 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:25:57.711892 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52a7fed_385d_45b9_a1a4_78be7a912e60.slice/crio-a38fc23028b443babc9c4b94cad316dfe8f8f4a162c6f43a6c6fc4ee67a75f0a WatchSource:0}: Error finding container a38fc23028b443babc9c4b94cad316dfe8f8f4a162c6f43a6c6fc4ee67a75f0a: Status 404 returned error can't find the container with id a38fc23028b443babc9c4b94cad316dfe8f8f4a162c6f43a6c6fc4ee67a75f0a Oct 30 19:25:58.280004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:25:58.279963 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" event={"ID":"c52a7fed-385d-45b9-a1a4-78be7a912e60","Type":"ContainerStarted","Data":"a38fc23028b443babc9c4b94cad316dfe8f8f4a162c6f43a6c6fc4ee67a75f0a"} Oct 30 19:26:00.295063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:00.295022 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" event={"ID":"c52a7fed-385d-45b9-a1a4-78be7a912e60","Type":"ContainerStarted","Data":"44adb2bdd3251ec9d13ae37ef07c4e0f96e85010786d42025064b9edc6b71623"} Oct 30 19:26:00.315833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:00.315786 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" podStartSLOduration=1.220364769 podStartE2EDuration="3.315770174s" totalImagesPullingTime="2.095405405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:00.314150291 +0000 UTC m=+366.290376248" watchObservedRunningTime="2026-10-30 19:26:00.315770174 +0000 UTC m=+366.291996117" Oct 30 19:26:00.423239 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:00.423203 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:26:01.039502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:01.039458 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:26:07.422921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:07.422830 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:26:07.435532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:07.435499 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:26:07.735282 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:26:07.735246 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:26:08.347180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:08.347142 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w" Oct 30 19:26:13.537567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.537532 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:13.540928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.540902 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.547849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.547816 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:26:13.548942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.548705 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:26:13.548942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.548816 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-czrts\"" Oct 30 19:26:13.550984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.550962 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:13.620348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.620312 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.620536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.620436 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr74n\" (UniqueName: \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.721153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.721114 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr74n\" (UniqueName: \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.721324 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.721217 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.723695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.723674 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.731056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.731032 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr74n\" (UniqueName: \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\") pod \"ai-gateway-controller-84dff4756b-2wr5j\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:13.857621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:13.857526 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:14.133974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:14.133883 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:14.137100 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:14.137065 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950c4e58_659e_4219_a1bd_563d9ad2dd70.slice/crio-77d7db64013c1dc896a7bdba9e3270c50def96f4a38083bfa2bc74853725e2c2 WatchSource:0}: Error finding container 77d7db64013c1dc896a7bdba9e3270c50def96f4a38083bfa2bc74853725e2c2: Status 404 returned error can't find the container with id 77d7db64013c1dc896a7bdba9e3270c50def96f4a38083bfa2bc74853725e2c2 Oct 30 19:26:14.367246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:14.367212 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" event={"ID":"950c4e58-659e-4219-a1bd-563d9ad2dd70","Type":"ContainerStarted","Data":"77d7db64013c1dc896a7bdba9e3270c50def96f4a38083bfa2bc74853725e2c2"} Oct 30 19:26:17.390143 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.390098 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" event={"ID":"950c4e58-659e-4219-a1bd-563d9ad2dd70","Type":"ContainerStarted","Data":"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779"} Oct 30 19:26:17.390661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.390286 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:17.409357 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.409240 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" podStartSLOduration=1.395330848 podStartE2EDuration="4.409225539s" totalImagesPullingTime="3.013894691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:17.406814056 +0000 UTC m=+383.383040002" watchObservedRunningTime="2026-10-30 19:26:17.409225539 +0000 UTC m=+383.385451494" Oct 30 19:26:17.925392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.925364 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh"] Oct 30 19:26:17.928984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.928958 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:17.938861 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.938831 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:26:17.942930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.942904 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pp54j"] Oct 30 19:26:17.946343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.946322 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh"] Oct 30 19:26:17.946517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.946408 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:17.956128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.955640 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r6lnq\"" Oct 30 19:26:17.958918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:17.958891 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pp54j"] Oct 30 19:26:18.063669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.063640 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a76f54a-2ca7-401b-86a0-7e51f33fced3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.063854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.063690 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01ba900-adf8-476f-9d76-08ac8ec45ca0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.063854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.063814 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlntv\" (UniqueName: \"kubernetes.io/projected/0a76f54a-2ca7-401b-86a0-7e51f33fced3-kube-api-access-wlntv\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.063948 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.063885 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qk8\" (UniqueName: \"kubernetes.io/projected/a01ba900-adf8-476f-9d76-08ac8ec45ca0-kube-api-access-l2qk8\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.165144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165110 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01ba900-adf8-476f-9d76-08ac8ec45ca0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.165328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165196 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlntv\" (UniqueName: \"kubernetes.io/projected/0a76f54a-2ca7-401b-86a0-7e51f33fced3-kube-api-access-wlntv\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.165328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165241 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qk8\" (UniqueName: \"kubernetes.io/projected/a01ba900-adf8-476f-9d76-08ac8ec45ca0-kube-api-access-l2qk8\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.165401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165368 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a76f54a-2ca7-401b-86a0-7e51f33fced3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.165974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165953 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01ba900-adf8-476f-9d76-08ac8ec45ca0-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.166070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.165994 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a76f54a-2ca7-401b-86a0-7e51f33fced3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.175223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.175198 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qk8\" (UniqueName: \"kubernetes.io/projected/a01ba900-adf8-476f-9d76-08ac8ec45ca0-kube-api-access-l2qk8\") pod \"payload-processing-55fbbbf9dc-pp54j\" (UID: \"a01ba900-adf8-476f-9d76-08ac8ec45ca0\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.175328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.175222 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlntv\" (UniqueName: \"kubernetes.io/projected/0a76f54a-2ca7-401b-86a0-7e51f33fced3-kube-api-access-wlntv\") pod \"payload-pre-processing-7b98bb678c-ddzlh\" (UID: \"0a76f54a-2ca7-401b-86a0-7e51f33fced3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.244034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.243996 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:18.259545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.259519 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:18.526533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.526508 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh"] Oct 30 19:26:18.529265 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:18.529235 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a76f54a_2ca7_401b_86a0_7e51f33fced3.slice/crio-5b090095c441b6fa367dd3a97e23bf72a4135e13f0179ec1e4d8e6b9bce01e9a WatchSource:0}: Error finding container 5b090095c441b6fa367dd3a97e23bf72a4135e13f0179ec1e4d8e6b9bce01e9a: Status 404 returned error can't find the container with id 5b090095c441b6fa367dd3a97e23bf72a4135e13f0179ec1e4d8e6b9bce01e9a Oct 30 19:26:18.549635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:18.549603 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-pp54j"] Oct 30 19:26:18.553340 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:18.553305 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01ba900_adf8_476f_9d76_08ac8ec45ca0.slice/crio-65c9b0dc8370e553125ec25cc3535b713ddd6d92a61c0414e476c91d9a31f845 WatchSource:0}: Error finding container 65c9b0dc8370e553125ec25cc3535b713ddd6d92a61c0414e476c91d9a31f845: Status 404 returned error can't find the container with id 65c9b0dc8370e553125ec25cc3535b713ddd6d92a61c0414e476c91d9a31f845 Oct 30 19:26:19.402687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:19.402643 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" event={"ID":"a01ba900-adf8-476f-9d76-08ac8ec45ca0","Type":"ContainerStarted","Data":"65c9b0dc8370e553125ec25cc3535b713ddd6d92a61c0414e476c91d9a31f845"} Oct 30 19:26:19.404842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:19.404810 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" event={"ID":"0a76f54a-2ca7-401b-86a0-7e51f33fced3","Type":"ContainerStarted","Data":"5b090095c441b6fa367dd3a97e23bf72a4135e13f0179ec1e4d8e6b9bce01e9a"} Oct 30 19:26:21.238501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.238463 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g"] Oct 30 19:26:21.243011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.242985 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.255065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.255037 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g"] Oct 30 19:26:21.298965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.298931 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.298965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.298978 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299198 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299081 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dthz\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-kube-api-access-8dthz\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299198 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299119 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299191 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299238 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299327 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299353 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.299390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.299384 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400251 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400303 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400336 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dthz\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-kube-api-access-8dthz\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400359 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400387 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400412 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400438 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400497 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400549 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.400879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.400798 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.401116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.401083 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.401247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.401089 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.401247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.401164 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.401494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.401473 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.402825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.402805 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.402923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.402879 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.410055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.410026 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.410149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.410105 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dthz\" (UniqueName: \"kubernetes.io/projected/3a9a5b0e-97f2-495a-89e7-516b96f2887f-kube-api-access-8dthz\") pod \"maas-default-gateway-openshift-default-75bf867b6c-pns2g\" (UID: \"3a9a5b0e-97f2-495a-89e7-516b96f2887f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.419227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.419197 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" event={"ID":"a01ba900-adf8-476f-9d76-08ac8ec45ca0","Type":"ContainerStarted","Data":"e1d1246f4cdbb0e8652d9bfa7f523b5234a5aacaae7ce531715ee6d9b8038e02"} Oct 30 19:26:21.419398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.419380 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:21.420931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.420906 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" event={"ID":"0a76f54a-2ca7-401b-86a0-7e51f33fced3","Type":"ContainerStarted","Data":"7ac53f302b7f1e630ce6fb43d3cb443d79379b7cc69e8b0e8f86454b8c4ccec7"} Oct 30 19:26:21.421044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.420970 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:21.436093 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.436049 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" podStartSLOduration=2.137321981 podStartE2EDuration="4.436036763s" totalImagesPullingTime="2.298714782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:21.434849266 +0000 UTC m=+387.411075210" watchObservedRunningTime="2026-10-30 19:26:21.436036763 +0000 UTC m=+387.412262706" Oct 30 19:26:21.451894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.451852 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" podStartSLOduration=2.177296529 podStartE2EDuration="4.451832441s" totalImagesPullingTime="2.274535912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:21.450512789 +0000 UTC m=+387.426738735" watchObservedRunningTime="2026-10-30 19:26:21.451832441 +0000 UTC m=+387.428058385" Oct 30 19:26:21.559501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.559393 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:21.837240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.837215 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g"] Oct 30 19:26:21.839735 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:21.839702 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9a5b0e_97f2_495a_89e7_516b96f2887f.slice/crio-d5c1e88e9fecfdcc4d8191acdb4362fe9fed5efff36b2846cce3dc29e17dad70 WatchSource:0}: Error finding container d5c1e88e9fecfdcc4d8191acdb4362fe9fed5efff36b2846cce3dc29e17dad70: Status 404 returned error can't find the container with id d5c1e88e9fecfdcc4d8191acdb4362fe9fed5efff36b2846cce3dc29e17dad70 Oct 30 19:26:21.842133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.842104 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:21.842207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.842175 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:21.842243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:21.842205 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:22.446481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.445920 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" event={"ID":"3a9a5b0e-97f2-495a-89e7-516b96f2887f","Type":"ContainerStarted","Data":"d581ab2c865d58a1dc68010f709c9d5e7d2a3137610cde90ead11d1ab9123918"} Oct 30 19:26:22.446481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.445982 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" event={"ID":"3a9a5b0e-97f2-495a-89e7-516b96f2887f","Type":"ContainerStarted","Data":"d5c1e88e9fecfdcc4d8191acdb4362fe9fed5efff36b2846cce3dc29e17dad70"} Oct 30 19:26:22.467886 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.467806 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podStartSLOduration=1.46778762 podStartE2EDuration="1.46778762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:22.466868945 +0000 UTC m=+388.443094891" watchObservedRunningTime="2026-10-30 19:26:22.46778762 +0000 UTC m=+388.444013565" Oct 30 19:26:22.560011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.559972 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:22.561430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.561406 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:22.561512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:22.561485 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:23.560414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:23.560377 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:23.560837 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:23.560435 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:24.560016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:24.559979 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:24.560206 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:24.560036 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:25.560487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:25.560433 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:25.560941 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:25.560505 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:26.560583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:26.560544 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:26.560983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:26.560605 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:27.560261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:27.560220 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:27.560476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:27.560282 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:28.397630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.397598 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:28.560253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.560219 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:26:28.560433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.560289 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 19:26:28.667159 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.667079 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-689bf9d68c-8qzns"] Oct 30 19:26:28.671069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.671048 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.686212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.686182 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-689bf9d68c-8qzns"] Oct 30 19:26:28.795301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.795247 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr68j\" (UniqueName: \"kubernetes.io/projected/303ef8a0-c0c7-4749-a661-6f95f42348c3-kube-api-access-dr68j\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.795533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.795325 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303ef8a0-c0c7-4749-a661-6f95f42348c3-metrics-certs\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.896273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.896223 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr68j\" (UniqueName: \"kubernetes.io/projected/303ef8a0-c0c7-4749-a661-6f95f42348c3-kube-api-access-dr68j\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.896479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.896286 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303ef8a0-c0c7-4749-a661-6f95f42348c3-metrics-certs\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.898703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.898674 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/303ef8a0-c0c7-4749-a661-6f95f42348c3-metrics-certs\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.905251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.905224 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr68j\" (UniqueName: \"kubernetes.io/projected/303ef8a0-c0c7-4749-a661-6f95f42348c3-kube-api-access-dr68j\") pod \"ai-gateway-controller-689bf9d68c-8qzns\" (UID: \"303ef8a0-c0c7-4749-a661-6f95f42348c3\") " pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:28.985977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:28.985943 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:29.262383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:29.262355 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-689bf9d68c-8qzns"] Oct 30 19:26:29.264595 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:29.264559 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303ef8a0_c0c7_4749_a661_6f95f42348c3.slice/crio-061f676f367b14ba8d4ce3752c92761eccac224e822734450198ba3cb70b32d5 WatchSource:0}: Error finding container 061f676f367b14ba8d4ce3752c92761eccac224e822734450198ba3cb70b32d5: Status 404 returned error can't find the container with id 061f676f367b14ba8d4ce3752c92761eccac224e822734450198ba3cb70b32d5 Oct 30 19:26:29.490661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:29.490621 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" event={"ID":"303ef8a0-c0c7-4749-a661-6f95f42348c3","Type":"ContainerStarted","Data":"061f676f367b14ba8d4ce3752c92761eccac224e822734450198ba3cb70b32d5"} Oct 30 19:26:30.339424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.339390 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:30.339893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.339865 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:30.340642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.340621 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" Oct 30 19:26:30.497272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.497232 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" event={"ID":"303ef8a0-c0c7-4749-a661-6f95f42348c3","Type":"ContainerStarted","Data":"4acb38f056e4df8d761139459ba3a769dddc5458bd7b77d78a2354277b60e338"} Oct 30 19:26:30.497793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.497316 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:30.515915 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:30.515864 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" podStartSLOduration=2.09556702 podStartE2EDuration="2.515851507s" totalImagesPullingTime="420.284487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:30.514527725 +0000 UTC m=+396.490753669" watchObservedRunningTime="2026-10-30 19:26:30.515851507 +0000 UTC m=+396.492077447" Oct 30 19:26:32.448025 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:32.447991 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-pp54j" Oct 30 19:26:32.453750 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:32.453725 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" Oct 30 19:26:36.263650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.263616 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7"] Oct 30 19:26:36.268002 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.267978 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.282979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.282946 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7"] Oct 30 19:26:36.369343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369305 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369348 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369376 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369526 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369556 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369596 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369638 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369699 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.369857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.369761 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v78w\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-kube-api-access-4v78w\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.470964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.470928 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.470986 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471033 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471059 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v78w\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-kube-api-access-4v78w\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471130 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471157 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471186 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471249 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471275 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471529 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471674 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471718 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471871 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471825 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.471925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.471914 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.473389 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.473366 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.473748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.473726 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.479982 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.479960 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.480162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.480138 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v78w\" (UniqueName: \"kubernetes.io/projected/0818d712-9d8b-4db5-9b05-ffca59cfdea2-kube-api-access-4v78w\") pod \"maas-default-gateway-openshift-default-75bf867b6c-vzsw7\" (UID: \"0818d712-9d8b-4db5-9b05-ffca59cfdea2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.587399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.587301 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:36.867255 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.867168 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7"] Oct 30 19:26:36.870397 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:36.870366 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0818d712_9d8b_4db5_9b05_ffca59cfdea2.slice/crio-10a0fb89ca61b42e1b4919f16ed1e31bdb120386fe550788cc06e3528b9b41b1 WatchSource:0}: Error finding container 10a0fb89ca61b42e1b4919f16ed1e31bdb120386fe550788cc06e3528b9b41b1: Status 404 returned error can't find the container with id 10a0fb89ca61b42e1b4919f16ed1e31bdb120386fe550788cc06e3528b9b41b1 Oct 30 19:26:36.872975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.872937 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:36.873074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.873020 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:36.873074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:36.873066 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:37.553784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.553739 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" event={"ID":"0818d712-9d8b-4db5-9b05-ffca59cfdea2","Type":"ContainerStarted","Data":"70150b58d663f4d6e0f98edd4cd9000bd97fc5af7b3579606d55e62e70ed3b88"} Oct 30 19:26:37.554203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.553805 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" event={"ID":"0818d712-9d8b-4db5-9b05-ffca59cfdea2","Type":"ContainerStarted","Data":"10a0fb89ca61b42e1b4919f16ed1e31bdb120386fe550788cc06e3528b9b41b1"} Oct 30 19:26:37.578845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.578785 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podStartSLOduration=1.578765347 podStartE2EDuration="1.578765347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:37.577795004 +0000 UTC m=+403.554020951" watchObservedRunningTime="2026-10-30 19:26:37.578765347 +0000 UTC m=+403.554991294" Oct 30 19:26:37.588190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.588161 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:37.589725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.589700 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:37.589845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:37.589761 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:37.759758 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:26:37.759716 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:26:38.588100 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:38.588066 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:38.588528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:38.588121 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:39.588022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:39.587984 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:39.588205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:39.588036 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:40.587829 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:40.587790 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:40.587992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:40.587853 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:41.504569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.504540 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-689bf9d68c-8qzns" Oct 30 19:26:41.552981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.552938 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:41.553351 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.553318 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" podUID="950c4e58-659e-4219-a1bd-563d9ad2dd70" containerName="manager" containerID="cri-o://c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779" gracePeriod=30 Oct 30 19:26:41.588077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.588050 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:41.588212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.588099 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:41.860361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.860330 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:41.951790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.951741 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\") pod \"950c4e58-659e-4219-a1bd-563d9ad2dd70\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " Oct 30 19:26:41.952015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.951812 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\" (UniqueName: \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\") pod \"950c4e58-659e-4219-a1bd-563d9ad2dd70\" (UID: \"950c4e58-659e-4219-a1bd-563d9ad2dd70\") " Oct 30 19:26:41.954081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.954050 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs" pod "950c4e58-659e-4219-a1bd-563d9ad2dd70" (UID: "950c4e58-659e-4219-a1bd-563d9ad2dd70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:41.954198 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:41.954052 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n" pod "950c4e58-659e-4219-a1bd-563d9ad2dd70" (UID: "950c4e58-659e-4219-a1bd-563d9ad2dd70"). InnerVolumeSpecName "kube-api-access-dr74n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:42.053120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.053084 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr74n\" (UniqueName: \"kubernetes.io/projected/950c4e58-659e-4219-a1bd-563d9ad2dd70-kube-api-access-dr74n\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:42.053120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.053117 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/950c4e58-659e-4219-a1bd-563d9ad2dd70-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:42.584259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.584219 6907 generic.go:356] "Generic (PLEG): container finished" podID="950c4e58-659e-4219-a1bd-563d9ad2dd70" containerID="c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779" exitCode=0 Oct 30 19:26:42.584742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.584283 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" Oct 30 19:26:42.584742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.584306 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" event={"ID":"950c4e58-659e-4219-a1bd-563d9ad2dd70","Type":"ContainerDied","Data":"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779"} Oct 30 19:26:42.584742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.584385 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-2wr5j" event={"ID":"950c4e58-659e-4219-a1bd-563d9ad2dd70","Type":"ContainerDied","Data":"77d7db64013c1dc896a7bdba9e3270c50def96f4a38083bfa2bc74853725e2c2"} Oct 30 19:26:42.584742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.584411 6907 scope.go:122] "RemoveContainer" containerID="c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779" Oct 30 19:26:42.587868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.587835 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:42.587992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.587884 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:42.594392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.594375 6907 scope.go:122] "RemoveContainer" containerID="c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779" Oct 30 19:26:42.594725 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:26:42.594704 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779\": container with ID starting with c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779 not found: ID does not exist" containerID="c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779" Oct 30 19:26:42.594807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.594736 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779"} err="failed to get container status \"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779\": rpc error: code = NotFound desc = could not find container \"c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779\": container with ID starting with c640c112ee8ea3674e6127976346d57bb39a7825df2a40a4ccb9966a9e1d7779 not found: ID does not exist" Oct 30 19:26:42.603717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.603686 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:42.616151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:42.616126 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-2wr5j"] Oct 30 19:26:43.588036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:43.588004 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:43.588406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:43.588059 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:44.150605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:44.150563 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950c4e58-659e-4219-a1bd-563d9ad2dd70" path="/var/lib/kubelet/pods/950c4e58-659e-4219-a1bd-563d9ad2dd70/volumes" Oct 30 19:26:44.587744 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:44.587708 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 19:26:44.588026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:44.587763 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 19:26:45.591887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:45.591846 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:45.592346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:45.592155 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:45.592958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:45.592941 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" Oct 30 19:26:51.287183 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.287140 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd"] Oct 30 19:26:51.287900 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.287869 6907 cpu_manager.go:414] "containerMap: removing container" podUID="950c4e58-659e-4219-a1bd-563d9ad2dd70" containerName="manager" Oct 30 19:26:51.288083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.288059 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950c4e58-659e-4219-a1bd-563d9ad2dd70" containerName="manager" Oct 30 19:26:51.288243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.288228 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="950c4e58-659e-4219-a1bd-563d9ad2dd70" containerName="manager" Oct 30 19:26:51.295622 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.295595 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.305797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.305773 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd"] Oct 30 19:26:51.448232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448192 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjstm\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-kube-api-access-zjstm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448236 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448260 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147200da-9ad7-4a69-b764-e074edb12c9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448386 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147200da-9ad7-4a69-b764-e074edb12c9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448556 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448501 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448575 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448617 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448639 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.448769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.448688 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549467 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549516 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549546 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549577 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549604 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549664 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjstm\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-kube-api-access-zjstm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549692 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549722 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147200da-9ad7-4a69-b764-e074edb12c9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.549854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.549750 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147200da-9ad7-4a69-b764-e074edb12c9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.550165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.550066 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-data\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.550165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.550107 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-credential-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.550439 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.550413 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-certs\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.550554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.550427 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-workload-socket\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.550554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.550498 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/147200da-9ad7-4a69-b764-e074edb12c9f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.552244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.552214 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/147200da-9ad7-4a69-b764-e074edb12c9f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.552558 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.552539 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/147200da-9ad7-4a69-b764-e074edb12c9f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.557973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.557952 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjstm\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-kube-api-access-zjstm\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.558056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.557975 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/147200da-9ad7-4a69-b764-e074edb12c9f-istio-token\") pod \"maas-default-gateway-openshift-default-75bf867b6c-rp9rd\" (UID: \"147200da-9ad7-4a69-b764-e074edb12c9f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.610796 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.610765 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:51.890142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.890114 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd"] Oct 30 19:26:51.892104 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:26:51.892073 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147200da_9ad7_4a69_b764_e074edb12c9f.slice/crio-70c04c13cb8d7a67cb65da0211fdf48dce5fa75ef9e47646a1b5d23008961eaf WatchSource:0}: Error finding container 70c04c13cb8d7a67cb65da0211fdf48dce5fa75ef9e47646a1b5d23008961eaf: Status 404 returned error can't find the container with id 70c04c13cb8d7a67cb65da0211fdf48dce5fa75ef9e47646a1b5d23008961eaf Oct 30 19:26:51.894349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.894317 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:51.894546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.894387 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:51.894546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:51.894418 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:26:52.643485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:52.642908 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" event={"ID":"147200da-9ad7-4a69-b764-e074edb12c9f","Type":"ContainerStarted","Data":"e4933c9b486ec7ddcb8f270e9ff99b1491c0f6aa0fb907470ed0e9e6a8fa60c7"} Oct 30 19:26:52.643485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:52.642984 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" event={"ID":"147200da-9ad7-4a69-b764-e074edb12c9f","Type":"ContainerStarted","Data":"70c04c13cb8d7a67cb65da0211fdf48dce5fa75ef9e47646a1b5d23008961eaf"} Oct 30 19:26:52.666655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:52.666597 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podStartSLOduration=1.666579493 podStartE2EDuration="1.666579493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:52.665262977 +0000 UTC m=+418.641488922" watchObservedRunningTime="2026-10-30 19:26:52.666579493 +0000 UTC m=+418.642805438" Oct 30 19:26:53.611162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:53.611117 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:26:53.612757 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:53.612721 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:53.612904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:53.612787 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:26:54.612017 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:54.611980 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:54.612422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:54.612033 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:26:55.611951 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:55.611915 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:55.612127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:55.611967 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:26:56.611486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:56.611423 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:56.611678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:56.611516 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:26:57.611582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:57.611548 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:57.612005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:57.611605 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:26:58.611764 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:58.611721 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 19:26:58.612178 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:26:58.611799 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 19:27:00.126152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:00.126112 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:27:00.126628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:00.126472 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:27:00.127510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:00.127487 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" Oct 30 19:27:07.736097 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:27:07.736056 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:27:27.574577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.574540 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:27:27.578857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.578831 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.587386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.587364 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3bc085-openshift-default-dockercfg-6wbgq\"" Oct 30 19:27:27.594200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.594167 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:27:27.719772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.719734 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.719772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.719772 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.719981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.719921 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.719981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.719943 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.720054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.720013 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5lh\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.720054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.720051 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.720117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.720072 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.720192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.720175 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.720233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.720221 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.820940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.820899 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.820952 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821014 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5lh\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821202 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821271 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821253 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821318 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821321 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821416 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821488 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821513 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821750 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.821994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.821968 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.822146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.822048 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.822295 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.822275 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.824084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.824062 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.824314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.824292 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.834667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.834586 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5lh\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.836763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.836732 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:27.864092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.862624 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:27:27.868182 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.868158 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:27.879887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.879802 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:27:27.886772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.886746 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-171808-openshift-default-dockercfg-mzssz\"" Oct 30 19:27:27.893962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:27.893936 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:28.023633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023600 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.023798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023659 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.023879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023845 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.023931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023899 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.023964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023942 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.024004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.023968 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.024104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.024083 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjlhb\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.024143 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.024128 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.024205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.024189 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.125836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.125749 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.125836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.125811 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.126082 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.125887 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.126141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126093 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.126191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126141 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.126191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126169 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126223 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126241 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjlhb\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126291 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126344 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126738 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126754 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.126897 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.127625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.127272 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.128732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.128706 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.129168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.129150 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.136869 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.136843 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjlhb\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.138648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.138623 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.185631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.185599 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:28.216062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.216030 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:27:28.221193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.221152 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.222145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.222115 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:27:28.224679 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:28.224582 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf26f935_3edf_458a_a38a_7a0ce517d86b.slice/crio-e6c4f417c88193b080fb5c5a5df6aa224fe414c678914eb95913aafac9948f4f WatchSource:0}: Error finding container e6c4f417c88193b080fb5c5a5df6aa224fe414c678914eb95913aafac9948f4f: Status 404 returned error can't find the container with id e6c4f417c88193b080fb5c5a5df6aa224fe414c678914eb95913aafac9948f4f Oct 30 19:27:28.226538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.226471 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f2d57-openshift-default-dockercfg-l4n9q\"" Oct 30 19:27:28.227931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.227898 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.228020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.227976 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.228059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.228022 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.236804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.234334 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328137 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328366 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328576 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328607 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328656 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328681 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328725 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328761 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.329072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.328799 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w22l\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.429998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.429904 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.430652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430615 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.430795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430665 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.430795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430702 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.430795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430729 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.430795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430774 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.431005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430808 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.431005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430846 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w22l\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.431005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430879 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.431274 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.431249 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.431580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.431554 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.432036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.430619 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.432036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.431801 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.432698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.432678 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.436880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.436826 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.437683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.437661 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.442288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.442179 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.442288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.442267 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w22l\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.484677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.484644 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:27:28.487706 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:28.487675 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784acf91_2cc1_4789_949e_faf0ffd6a82b.slice/crio-bb0bd7b41d38170c043c06301e39c1737bdcc136deab9c1cd2dce08275adfadd WatchSource:0}: Error finding container bb0bd7b41d38170c043c06301e39c1737bdcc136deab9c1cd2dce08275adfadd: Status 404 returned error can't find the container with id bb0bd7b41d38170c043c06301e39c1737bdcc136deab9c1cd2dce08275adfadd Oct 30 19:27:28.491696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.491661 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.491774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.491730 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.491774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.491759 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.547401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.547369 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:28.856347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.848790 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:27:28.857094 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:28.856617 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e109c52_c2fd_49b8_9172_374d5b43f3ea.slice/crio-3c8f65d36b1600ae5e89e25b3721feb40fdbc76ce98d245ed2ab211eea58ccbb WatchSource:0}: Error finding container 3c8f65d36b1600ae5e89e25b3721feb40fdbc76ce98d245ed2ab211eea58ccbb: Status 404 returned error can't find the container with id 3c8f65d36b1600ae5e89e25b3721feb40fdbc76ce98d245ed2ab211eea58ccbb Oct 30 19:27:28.859051 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.859024 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerStarted","Data":"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113"} Oct 30 19:27:28.859151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.859081 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerStarted","Data":"bb0bd7b41d38170c043c06301e39c1737bdcc136deab9c1cd2dce08275adfadd"} Oct 30 19:27:28.860091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.860058 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.860201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.860142 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.860201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.860187 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:28.865226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.864671 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerStarted","Data":"ae4e54917fb5fe236a247d119acf87a97978c5f76168acd5ef05eb283dad8c9d"} Oct 30 19:27:28.865226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.864778 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerStarted","Data":"e6c4f417c88193b080fb5c5a5df6aa224fe414c678914eb95913aafac9948f4f"} Oct 30 19:27:28.881813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.881767 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podStartSLOduration=1.881753098 podStartE2EDuration="1.881753098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:28.880136661 +0000 UTC m=+454.856362606" watchObservedRunningTime="2026-10-30 19:27:28.881753098 +0000 UTC m=+454.857979043" Oct 30 19:27:28.894073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.894048 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:28.903155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:28.903103 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podStartSLOduration=1.90306067 podStartE2EDuration="1.90306067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:28.901241937 +0000 UTC m=+454.877467880" watchObservedRunningTime="2026-10-30 19:27:28.90306067 +0000 UTC m=+454.879286616" Oct 30 19:27:29.186235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.186139 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:29.192359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.192330 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:29.872072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.872025 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerStarted","Data":"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f"} Oct 30 19:27:29.872636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.872089 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerStarted","Data":"3c8f65d36b1600ae5e89e25b3721feb40fdbc76ce98d245ed2ab211eea58ccbb"} Oct 30 19:27:29.872636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.872422 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:29.873357 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.873341 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:29.897905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.897772 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podStartSLOduration=1.897753046 podStartE2EDuration="1.897753046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:29.897522362 +0000 UTC m=+455.873748307" watchObservedRunningTime="2026-10-30 19:27:29.897753046 +0000 UTC m=+455.873978994" Oct 30 19:27:29.900707 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:29.900678 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:30.547711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.547678 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:30.553139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.553105 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:30.878099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.878019 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:30.878099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.878074 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:30.878674 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.878657 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:27:30.878726 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:30.878706 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:32.814069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.814038 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:27:32.817850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.817829 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:32.826630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.826605 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3bc085\"" Oct 30 19:27:32.827703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.827675 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3bc085-dockercfg-b6l4v\"" Oct 30 19:27:32.830331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.830307 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:27:32.884462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.884407 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:32.884644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.884565 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4r9\" (UniqueName: \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:32.988466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.986674 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4r9\" (UniqueName: \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:32.988466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.986847 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:32.988466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:32.987891 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:33.014179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.014141 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4r9\" (UniqueName: \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\") pod \"payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:33.134570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.134482 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:33.430806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.430774 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:27:33.431233 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:33.431202 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be55339_a855_4dab_89a2_2abac9065974.slice/crio-913a67e4a009f6647b1f441d2216b33d269ab5017f82965a088d9361101e7610 WatchSource:0}: Error finding container 913a67e4a009f6647b1f441d2216b33d269ab5017f82965a088d9361101e7610: Status 404 returned error can't find the container with id 913a67e4a009f6647b1f441d2216b33d269ab5017f82965a088d9361101e7610 Oct 30 19:27:33.870066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.870026 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:27:33.873821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.873802 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:33.882544 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.882513 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6cf459-dockercfg-g4hx4\"" Oct 30 19:27:33.887734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.887676 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6cf459\"" Oct 30 19:27:33.893716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.893681 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:27:33.894302 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.894269 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9drq\" (UniqueName: \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:33.894428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.894322 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:33.898482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.898411 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" event={"ID":"1be55339-a855-4dab-89a2-2abac9065974","Type":"ContainerStarted","Data":"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf"} Oct 30 19:27:33.898598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.898510 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" event={"ID":"1be55339-a855-4dab-89a2-2abac9065974","Type":"ContainerStarted","Data":"913a67e4a009f6647b1f441d2216b33d269ab5017f82965a088d9361101e7610"} Oct 30 19:27:33.898598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.898543 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:33.921644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.921580 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" podStartSLOduration=1.921560569 podStartE2EDuration="1.921560569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:33.920320433 +0000 UTC m=+459.896546377" watchObservedRunningTime="2026-10-30 19:27:33.921560569 +0000 UTC m=+459.897786514" Oct 30 19:27:33.995580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.995539 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9drq\" (UniqueName: \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:33.995580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.995580 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:33.996307 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:33.996286 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:34.005748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.005716 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9drq\" (UniqueName: \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\") pod \"payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:34.194365 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.194280 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:34.478799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.478767 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:27:34.481194 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:34.481165 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9ddcad_b84c_48eb_bf61_92a4dd1bf91f.slice/crio-1d411e7962ec79a46f22a114477ab58c775939b1cc0ada8586240f92fd15b915 WatchSource:0}: Error finding container 1d411e7962ec79a46f22a114477ab58c775939b1cc0ada8586240f92fd15b915: Status 404 returned error can't find the container with id 1d411e7962ec79a46f22a114477ab58c775939b1cc0ada8586240f92fd15b915 Oct 30 19:27:34.909971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.909865 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" event={"ID":"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f","Type":"ContainerStarted","Data":"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406"} Oct 30 19:27:34.909971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.909934 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" event={"ID":"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f","Type":"ContainerStarted","Data":"1d411e7962ec79a46f22a114477ab58c775939b1cc0ada8586240f92fd15b915"} Oct 30 19:27:34.910578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.910186 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:34.933387 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:34.933330 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" podStartSLOduration=1.93331491 podStartE2EDuration="1.93331491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:34.931230954 +0000 UTC m=+460.907456897" watchObservedRunningTime="2026-10-30 19:27:34.93331491 +0000 UTC m=+460.909540853" Oct 30 19:27:35.455298 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.455264 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:27:35.459125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.459109 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.474570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.474542 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d55178\"" Oct 30 19:27:35.475740 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.475715 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:27:35.509485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.509455 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:27:35.513149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.513128 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.518313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.518282 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.518424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.518335 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5vq\" (UniqueName: \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.518571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.518531 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.518653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.518637 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwpv\" (UniqueName: \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.525213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.525193 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d55178-dockercfg-th2wh\"" Oct 30 19:27:35.533852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.533820 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:27:35.639559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.638105 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.639559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.638204 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwpv\" (UniqueName: \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.639559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.638261 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.639559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.638299 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5vq\" (UniqueName: \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.640121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.639741 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.640121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.640077 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.686611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.686415 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwpv\" (UniqueName: \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\") pod \"payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.694130 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.694094 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5vq\" (UniqueName: \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\") pod \"payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:35.772425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.772274 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:35.828494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:35.827927 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:36.087410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.087335 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:27:36.091613 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:36.091582 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613fd7aa_051f_4807_9e5b_9d27a952d84d.slice/crio-5ed426a6b0023f45b6fcd3ffc1bd1824ac8e5b7d1601271945e9b625b015602d WatchSource:0}: Error finding container 5ed426a6b0023f45b6fcd3ffc1bd1824ac8e5b7d1601271945e9b625b015602d: Status 404 returned error can't find the container with id 5ed426a6b0023f45b6fcd3ffc1bd1824ac8e5b7d1601271945e9b625b015602d Oct 30 19:27:36.107268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.107234 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:27:36.113116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.113094 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.117516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.117497 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-639c15-dockercfg-2wzsf\"" Oct 30 19:27:36.117778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.117762 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-639c15\"" Oct 30 19:27:36.124187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.124166 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:27:36.134392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.134340 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:27:36.137502 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:36.137428 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db6bcb9_ee22_4636_a84c_64fbf9e9e1e0.slice/crio-c83613a86305308be8e457c65b594a257b3ac50aee48a931d6cf382973146245 WatchSource:0}: Error finding container c83613a86305308be8e457c65b594a257b3ac50aee48a931d6cf382973146245: Status 404 returned error can't find the container with id c83613a86305308be8e457c65b594a257b3ac50aee48a931d6cf382973146245 Oct 30 19:27:36.248252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.248205 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxlq\" (UniqueName: \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.264470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.263596 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.364567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.364454 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxlq\" (UniqueName: \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.364567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.364517 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.365488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.365427 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.375338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.375305 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxlq\" (UniqueName: \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\") pod \"payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.461401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.461364 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.740044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.740016 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:27:36.742261 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:36.742231 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fad9cf2_0d81_4dcf_9216_c85a3e32476b.slice/crio-db116370a3c5ec66abf363874c85162778a895043c5330a51927ca18cdcdd8e6 WatchSource:0}: Error finding container db116370a3c5ec66abf363874c85162778a895043c5330a51927ca18cdcdd8e6: Status 404 returned error can't find the container with id db116370a3c5ec66abf363874c85162778a895043c5330a51927ca18cdcdd8e6 Oct 30 19:27:36.924832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.924790 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" event={"ID":"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0","Type":"ContainerStarted","Data":"7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086"} Oct 30 19:27:36.925024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.924855 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" event={"ID":"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0","Type":"ContainerStarted","Data":"c83613a86305308be8e457c65b594a257b3ac50aee48a931d6cf382973146245"} Oct 30 19:27:36.925024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.924881 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:36.926546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.926517 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" event={"ID":"613fd7aa-051f-4807-9e5b-9d27a952d84d","Type":"ContainerStarted","Data":"7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc"} Oct 30 19:27:36.926666 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.926569 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" event={"ID":"613fd7aa-051f-4807-9e5b-9d27a952d84d","Type":"ContainerStarted","Data":"5ed426a6b0023f45b6fcd3ffc1bd1824ac8e5b7d1601271945e9b625b015602d"} Oct 30 19:27:36.926666 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.926596 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:36.928314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.928283 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" event={"ID":"4fad9cf2-0d81-4dcf-9216-c85a3e32476b","Type":"ContainerStarted","Data":"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe"} Oct 30 19:27:36.928476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.928337 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" event={"ID":"4fad9cf2-0d81-4dcf-9216-c85a3e32476b","Type":"ContainerStarted","Data":"db116370a3c5ec66abf363874c85162778a895043c5330a51927ca18cdcdd8e6"} Oct 30 19:27:36.928476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.928364 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:36.948261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.948206 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" podStartSLOduration=1.948191414 podStartE2EDuration="1.948191414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:36.945766076 +0000 UTC m=+462.921992041" watchObservedRunningTime="2026-10-30 19:27:36.948191414 +0000 UTC m=+462.924417358" Oct 30 19:27:36.969040 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.968975 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" podStartSLOduration=0.968957773 podStartE2EDuration="968.957773ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:36.968928141 +0000 UTC m=+462.945154080" watchObservedRunningTime="2026-10-30 19:27:36.968957773 +0000 UTC m=+462.945183719" Oct 30 19:27:36.996532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:36.993508 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" podStartSLOduration=1.993483836 podStartE2EDuration="1.993483836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:36.989396287 +0000 UTC m=+462.965622234" watchObservedRunningTime="2026-10-30 19:27:36.993483836 +0000 UTC m=+462.969709783" Oct 30 19:27:37.363037 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.362939 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:27:37.367114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.367089 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.375480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.374655 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:27:37.375480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.375242 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-f7f075\"" Oct 30 19:27:37.375679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.375668 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-f7f075-dockercfg-x4bh4\"" Oct 30 19:27:37.416333 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.416288 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:27:37.420725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.420703 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.435528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.435323 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:27:37.495478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.493243 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.495478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.493349 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4q5s\" (UniqueName: \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.596671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.594405 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4q5s\" (UniqueName: \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.596671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.594616 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.596671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.594646 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwkc8\" (UniqueName: \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.596671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.594703 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.596671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.595736 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.613456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.613358 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4q5s\" (UniqueName: \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\") pod \"payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.687165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.687122 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:37.699402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.696667 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.699402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.696716 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwkc8\" (UniqueName: \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.699402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.697843 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.708816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.708788 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwkc8\" (UniqueName: \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\") pod \"payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.744005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.743970 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:37.765281 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:27:37.765057 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:27:37.854402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.854367 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:27:37.859373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.859351 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:37.869616 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.869592 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:27:37.873683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.873644 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9f2d57-dockercfg-b6jpw\"" Oct 30 19:27:37.873812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.873753 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9f2d57\"" Oct 30 19:27:37.892339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:37.892306 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:27:38.001506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.001095 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.001506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.001249 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjbg\" (UniqueName: \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.102276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.102233 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.102480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.102385 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjbg\" (UniqueName: \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.103199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.103178 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.119213 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:38.119184 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bb7e43_bcc2_46c7_92ba_f0f63b9c1e93.slice/crio-ccc34dcba6574bf0387137f8666da3b1ec99ec8e09839ebc78eca63f7bb17f27 WatchSource:0}: Error finding container ccc34dcba6574bf0387137f8666da3b1ec99ec8e09839ebc78eca63f7bb17f27: Status 404 returned error can't find the container with id ccc34dcba6574bf0387137f8666da3b1ec99ec8e09839ebc78eca63f7bb17f27 Oct 30 19:27:38.122211 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:38.121343 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45daa901_2222_4562_82f0_e0752975b69f.slice/crio-8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf WatchSource:0}: Error finding container 8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf: Status 404 returned error can't find the container with id 8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf Oct 30 19:27:38.123633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.123610 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:27:38.126670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.126645 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:27:38.133282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.133255 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjbg\" (UniqueName: \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\") pod \"payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.177623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.177592 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.474507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.474474 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:27:38.478975 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:38.478948 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice/crio-0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9 WatchSource:0}: Error finding container 0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9: Status 404 returned error can't find the container with id 0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9 Oct 30 19:27:38.948530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.948004 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" event={"ID":"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93","Type":"ContainerStarted","Data":"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85"} Oct 30 19:27:38.948530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.948091 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" event={"ID":"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93","Type":"ContainerStarted","Data":"ccc34dcba6574bf0387137f8666da3b1ec99ec8e09839ebc78eca63f7bb17f27"} Oct 30 19:27:38.952492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.949119 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:38.952492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.951364 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" event={"ID":"d56c69d2-5016-4ff6-89c0-1e6962c2a17f","Type":"ContainerStarted","Data":"10c4a0ac81725ca7a4e38281e7f0230d688fc4e6138bb7dea1f2ef6cf957d2cb"} Oct 30 19:27:38.952492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.951419 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" event={"ID":"d56c69d2-5016-4ff6-89c0-1e6962c2a17f","Type":"ContainerStarted","Data":"0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9"} Oct 30 19:27:38.952492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.952163 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:38.955428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.955395 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" event={"ID":"45daa901-2222-4562-82f0-e0752975b69f","Type":"ContainerStarted","Data":"c4d16586d8d2de64cc6fd0d230553c413151236663722db28a045d8e3d21848d"} Oct 30 19:27:38.955580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.955490 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" event={"ID":"45daa901-2222-4562-82f0-e0752975b69f","Type":"ContainerStarted","Data":"8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf"} Oct 30 19:27:38.955580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.955523 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:38.983372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:38.983307 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" podStartSLOduration=1.983288238 podStartE2EDuration="1.983288238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:38.979160013 +0000 UTC m=+464.955385959" watchObservedRunningTime="2026-10-30 19:27:38.983288238 +0000 UTC m=+464.959514183" Oct 30 19:27:39.022418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:39.022364 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" podStartSLOduration=2.022349517 podStartE2EDuration="2.022349517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:39.020745339 +0000 UTC m=+464.996971283" watchObservedRunningTime="2026-10-30 19:27:39.022349517 +0000 UTC m=+464.998575460" Oct 30 19:27:39.051428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:39.051339 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" podStartSLOduration=2.05131738 podStartE2EDuration="2.05131738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:39.050847331 +0000 UTC m=+465.027073277" watchObservedRunningTime="2026-10-30 19:27:39.05131738 +0000 UTC m=+465.027543325" Oct 30 19:27:42.253258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:42.253214 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:27:43.226342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.226303 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:27:43.261917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.261882 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:27:43.262419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.262020 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.269097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.269068 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:27:43.269269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.269236 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-898sh\"" Oct 30 19:27:43.269507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.269489 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:27:43.269588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.269516 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9f2d57-serving-cert\"" Oct 30 19:27:43.269588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.269582 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:27:43.393800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.393763 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.394009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.393817 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.394009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.393961 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksl8\" (UniqueName: \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.394099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.394053 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.495687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.495583 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.495865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.495742 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.495865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.495785 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksl8\" (UniqueName: \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.495975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.495869 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.496519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.496496 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.498161 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.498123 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.498261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.498195 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.505378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.505355 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksl8\" (UniqueName: \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\") pod \"maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.583656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.583613 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:43.990146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.988892 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:27:43.990146 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:27:43.989710 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c99e4d_3238_438b_9288_83580c2f79ea.slice/crio-5d14ef2a4890cc927240461beb0184003c67dffbcf38eb82af434b8a249af556 WatchSource:0}: Error finding container 5d14ef2a4890cc927240461beb0184003c67dffbcf38eb82af434b8a249af556: Status 404 returned error can't find the container with id 5d14ef2a4890cc927240461beb0184003c67dffbcf38eb82af434b8a249af556 Oct 30 19:27:43.991215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:43.991192 6907 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:27:44.008192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:44.008152 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" event={"ID":"d6c99e4d-3238-438b-9288-83580c2f79ea","Type":"ContainerStarted","Data":"5d14ef2a4890cc927240461beb0184003c67dffbcf38eb82af434b8a249af556"} Oct 30 19:27:44.914185 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:44.914096 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:27:45.921183 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:45.921152 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:27:46.919659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:46.919607 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:27:46.919866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:46.919692 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:27:47.053986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.053944 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" event={"ID":"d6c99e4d-3238-438b-9288-83580c2f79ea","Type":"ContainerStarted","Data":"31facd49f39b853fdd2eaa9c9080e4f6dd214984c34c2cc8aa3b1c4e419400b5"} Oct 30 19:27:47.054518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.054065 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:47.076986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.076929 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" podStartSLOduration=2.14944708 podStartE2EDuration="4.07691352s" totalImagesPullingTime="1.92746644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:47.074218324 +0000 UTC m=+473.050444293" watchObservedRunningTime="2026-10-30 19:27:47.07691352 +0000 UTC m=+473.053139464" Oct 30 19:27:47.558132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.558088 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:27:47.954550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.953053 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:27:47.954550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.953486 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:27:47.954550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:47.954014 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:27:49.989407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:49.989380 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:27:50.090573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.090476 6907 generic.go:356] "Generic (PLEG): container finished" podID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerID="ae4e54917fb5fe236a247d119acf87a97978c5f76168acd5ef05eb283dad8c9d" exitCode=0 Oct 30 19:27:50.090573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.090549 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerDied","Data":"ae4e54917fb5fe236a247d119acf87a97978c5f76168acd5ef05eb283dad8c9d"} Oct 30 19:27:50.090977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.090957 6907 scope.go:122] "RemoveContainer" containerID="ae4e54917fb5fe236a247d119acf87a97978c5f76168acd5ef05eb283dad8c9d" Oct 30 19:27:50.092482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.092435 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:50.092585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.092547 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:50.092645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.092595 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:50.994134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.994091 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:27:50.994871 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:50.994854 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:27:51.103715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:51.103679 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerStarted","Data":"3d6c7096815444383927ac19bb876def50dd9eb9599a7db6a93a8d6e61bae26a"} Oct 30 19:27:51.894540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:51.894499 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:51.894880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:51.894858 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:51.894927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:51.894906 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 19:27:52.895366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:52.895331 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:52.895787 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:52.895404 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 19:27:53.087925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:53.087894 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:27:53.894925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:53.894881 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:53.895111 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:53.894939 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 19:27:54.895142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:54.895050 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:54.895142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:54.895121 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 19:27:55.894965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:55.894867 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 30 19:27:55.894965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:55.894945 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 30 19:27:56.162594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.162510 6907 generic.go:356] "Generic (PLEG): container finished" podID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerID="55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113" exitCode=0 Oct 30 19:27:56.162594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.162557 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerDied","Data":"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113"} Oct 30 19:27:56.162953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.162939 6907 scope.go:122] "RemoveContainer" containerID="55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113" Oct 30 19:27:56.164520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.164481 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:56.164618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.164575 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:56.164668 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.164617 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:27:56.899887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.899844 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:56.900383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.900340 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:56.901845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:56.901815 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:27:57.182032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:57.181936 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerStarted","Data":"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010"} Oct 30 19:27:57.186139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:57.186113 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:27:57.187491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:57.187468 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:57.187614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:57.187520 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:27:58.186742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:58.186711 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:58.187236 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:58.186764 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:27:58.248257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:58.245134 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:27:59.188572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:59.188488 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:27:59.191386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:59.189525 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:27:59.872587 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:27:59.872541 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:28:00.186145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.186053 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:28:00.186145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.186109 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:28:00.204534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.204499 6907 generic.go:356] "Generic (PLEG): container finished" podID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerID="922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f" exitCode=0 Oct 30 19:28:00.204909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.204570 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerDied","Data":"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f"} Oct 30 19:28:00.204963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.204949 6907 scope.go:122] "RemoveContainer" containerID="922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f" Oct 30 19:28:00.206331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.206303 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:00.206396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.206376 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:00.206431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.206410 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:00.879465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:00.878557 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:28:01.186314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.186221 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:28:01.186314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.186281 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:28:01.212733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.212701 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerStarted","Data":"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d"} Oct 30 19:28:01.551469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.549677 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:28:01.551469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.551066 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:01.551469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:01.551250 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:02.188036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:02.187994 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:28:02.188270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:02.188065 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:28:02.550017 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:02.549971 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:02.550678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:02.550628 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:03.191061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:03.191028 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:28:03.192103 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:03.192083 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:28:03.548469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:03.547980 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:03.548469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:03.548081 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:04.551941 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:04.551890 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:04.552480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:04.551958 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:05.548468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:05.547882 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:05.548468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:05.547951 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:06.548454 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:06.548403 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 19:28:06.548873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:06.548492 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 19:28:07.552696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:07.552663 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:28:07.553088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:07.552950 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:28:07.553751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:07.553731 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:28:07.741035 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:28:07.740989 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:28:10.833811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:10.833776 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:28:18.124614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.124580 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:28:18.129014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.128980 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.135832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.135809 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-171808\"/\"openshift-service-ca.crt\"" Oct 30 19:28:18.136053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.136028 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-171808\"/\"kube-root-ca.crt\"" Oct 30 19:28:18.136252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.136238 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-171808\"/\"premium-w0-171808-kserve-self-signed-certs\"" Oct 30 19:28:18.136620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.136603 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-171808\"/\"default-dockercfg-mtr5j\"" Oct 30 19:28:18.142055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.142027 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.278718 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.278772 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.278915 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.278964 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7q4s\" (UniqueName: \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.279007 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.280512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.279039 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380357 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380404 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380497 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380531 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7q4s\" (UniqueName: \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380572 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380602 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.380913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380856 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.381000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.380977 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.381056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.381027 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.382792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.382768 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.383060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.383043 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.389971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.389952 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7q4s\" (UniqueName: \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\") pod \"premium-w0-171808-kserve-fbbdbdd9c-7xnjm\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.445349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.445319 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:18.724368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:18.724338 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:28:18.726693 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:28:18.726666 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6b2591_c659_41ed_9afe_4509c9232b51.slice/crio-6b4821649f3c06a100122061571b3d6ea16a30819ae4f9c9988d1b2d1937ada0 WatchSource:0}: Error finding container 6b4821649f3c06a100122061571b3d6ea16a30819ae4f9c9988d1b2d1937ada0: Status 404 returned error can't find the container with id 6b4821649f3c06a100122061571b3d6ea16a30819ae4f9c9988d1b2d1937ada0 Oct 30 19:28:19.358126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:19.358092 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" event={"ID":"ee6b2591-c659-41ed-9afe-4509c9232b51","Type":"ContainerStarted","Data":"e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997"} Oct 30 19:28:19.358537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:19.358138 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" event={"ID":"ee6b2591-c659-41ed-9afe-4509c9232b51","Type":"ContainerStarted","Data":"6b4821649f3c06a100122061571b3d6ea16a30819ae4f9c9988d1b2d1937ada0"} Oct 30 19:28:19.393839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:19.393787 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podStartSLOduration=1.393771764 podStartE2EDuration="1.393771764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:19.390845911 +0000 UTC m=+505.367071856" watchObservedRunningTime="2026-10-30 19:28:19.393771764 +0000 UTC m=+505.369997702" Oct 30 19:28:19.446355 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:19.446324 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:23.067016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:23.066980 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:28:28.445926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:28.445883 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:28.459054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:28.459021 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:29.431658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.431623 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:28:29.452347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.452313 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:28:29.456855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.456832 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.471562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.471524 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"openshift-service-ca.crt\"" Oct 30 19:28:29.472572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.472547 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"kube-root-ca.crt\"" Oct 30 19:28:29.472786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.472607 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"default-dockercfg-vtw2b\"" Oct 30 19:28:29.472786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.472644 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"premium-w2-d55178-kserve-self-signed-certs\"" Oct 30 19:28:29.475744 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.475718 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:28:29.558832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.558797 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.559011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.558875 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.559011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.558940 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.559011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.558995 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.559189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.559049 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.559189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.559068 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9kvh\" (UniqueName: \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660473 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660584 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660618 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9kvh\" (UniqueName: \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660713 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660827 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660804 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660838 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.660966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660944 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.661032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.660994 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.661215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.661192 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.662913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.662896 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.663190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.663172 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.670034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.670010 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9kvh\" (UniqueName: \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\") pod \"premium-w2-d55178-kserve-6b77675f98-pkrb7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:29.777539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:29.777482 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:30.134898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:30.134872 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:28:30.136838 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:28:30.136808 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e49551_f63b_4de9_80b1_fd7f4b7283b7.slice/crio-9c6ac4ffcc722cc91faba4efb5ffc7aeff1496929277bf58c53bdba53a83b130 WatchSource:0}: Error finding container 9c6ac4ffcc722cc91faba4efb5ffc7aeff1496929277bf58c53bdba53a83b130: Status 404 returned error can't find the container with id 9c6ac4ffcc722cc91faba4efb5ffc7aeff1496929277bf58c53bdba53a83b130 Oct 30 19:28:30.427502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:30.427391 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" event={"ID":"92e49551-f63b-4de9-80b1-fd7f4b7283b7","Type":"ContainerStarted","Data":"9d0526b62d6c7547c1144fca37f0d56f74d2099c2d99702a9c11201e4aa0f761"} Oct 30 19:28:30.427502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:30.427467 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" event={"ID":"92e49551-f63b-4de9-80b1-fd7f4b7283b7","Type":"ContainerStarted","Data":"9c6ac4ffcc722cc91faba4efb5ffc7aeff1496929277bf58c53bdba53a83b130"} Oct 30 19:28:30.446437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:30.446375 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podStartSLOduration=1.446355915 podStartE2EDuration="1.446355915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:30.445191532 +0000 UTC m=+516.421417476" watchObservedRunningTime="2026-10-30 19:28:30.446355915 +0000 UTC m=+516.422581861" Oct 30 19:28:30.779782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:30.779742 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:37.735248 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:28:37.735204 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:28:38.156017 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:38.155924 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:28:39.780528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:39.780490 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:39.793633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:39.793604 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:40.517986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:40.517956 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:28:43.157508 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.157457 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:28:43.165717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.165688 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.175168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.175134 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:28:43.180578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.180548 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6cf459-openshift-default-dockercfg-ckp8r\"" Oct 30 19:28:43.261350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261319 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261391 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261502 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261542 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261591 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261610 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261652 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwb8n\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261730 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.261789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.261776 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.362968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.362934 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.362992 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363044 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363107 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363136 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363160 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwb8n\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363207 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363432 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363542 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363560 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363691 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.363809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.363796 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.364049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.364031 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.364158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.364135 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.365409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.365388 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.365660 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.365639 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.372535 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.372510 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwb8n\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.372643 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.372590 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.486273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.486235 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:43.786623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.786595 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:28:43.788202 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:28:43.788174 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956c90bd_7b71_44b1_9444_65dfe74e159b.slice/crio-5c7781f456563409d36495cf805301bce1fcf7ed5347b185a45bbdaa87914812 WatchSource:0}: Error finding container 5c7781f456563409d36495cf805301bce1fcf7ed5347b185a45bbdaa87914812: Status 404 returned error can't find the container with id 5c7781f456563409d36495cf805301bce1fcf7ed5347b185a45bbdaa87914812 Oct 30 19:28:43.790823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.790785 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:43.790925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.790879 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:43.790993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:43.790924 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:44.563590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:44.563552 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" event={"ID":"956c90bd-7b71-44b1-9444-65dfe74e159b","Type":"ContainerStarted","Data":"fa02f84a4f4ae53727dc94b5952ec763d12e3a28b7b0e28178d759711e04404c"} Oct 30 19:28:44.564030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:44.563606 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" event={"ID":"956c90bd-7b71-44b1-9444-65dfe74e159b","Type":"ContainerStarted","Data":"5c7781f456563409d36495cf805301bce1fcf7ed5347b185a45bbdaa87914812"} Oct 30 19:28:44.584745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:44.584694 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podStartSLOduration=1.584679413 podStartE2EDuration="1.584679413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:44.58267738 +0000 UTC m=+530.558903326" watchObservedRunningTime="2026-10-30 19:28:44.584679413 +0000 UTC m=+530.560905351" Oct 30 19:28:45.487196 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:45.487159 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:45.489096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:45.489069 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:28:45.489222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:45.489118 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:28:46.487128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:46.487092 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:28:46.487577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:46.487150 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:28:47.487035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:47.487000 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:28:47.487218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:47.487066 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:28:48.486841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:48.486805 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:28:48.487031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:48.486859 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:28:49.487338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:49.487300 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 19:28:49.487767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:49.487373 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 19:28:50.573392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:50.573359 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:50.573948 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:50.573745 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:50.574623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:50.574600 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:28:57.825120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.825080 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:28:57.830531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.830504 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.848571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.848545 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:28:57.876756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.876725 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f7f075-openshift-default-dockercfg-p74l2\"" Oct 30 19:28:57.920286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920249 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cs7\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920304 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920417 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920498 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920567 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920663 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920695 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920794 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:57.920850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:57.920809 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021604 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021641 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021684 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cs7\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021724 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021754 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.021863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021808 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.021866 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022027 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022211 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022100 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022211 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022145 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022215 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022342 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022591 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.022733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.022675 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.024005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.023982 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.024309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.024285 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.035716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.035684 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cs7\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.035824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.035781 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.143708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.143622 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:58.249175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.249137 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:28:58.254321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.254297 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.259232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.259207 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d55178-openshift-default-dockercfg-6lspn\"" Oct 30 19:28:58.268885 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.268846 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:28:58.325870 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.325829 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txmkm\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.325891 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.325994 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326082 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326103 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326150 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326260 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326322 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.326388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.326376 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428041 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428095 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428173 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428205 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428261 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428297 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428359 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428412 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428488 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txmkm\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428599 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428668 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428748 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.428926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.428904 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.429062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.429045 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.430548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.430520 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.430869 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.430849 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.441795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.441760 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.442145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.442125 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txmkm\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.446290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.446264 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:28:58.447202 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:28:58.447175 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a404769_ddd9_473d_8d14_bd668ced40bd.slice/crio-aa9693fc408d88b1418c500204b083be0cb3b3aa72c035b613229141a403ed23 WatchSource:0}: Error finding container aa9693fc408d88b1418c500204b083be0cb3b3aa72c035b613229141a403ed23: Status 404 returned error can't find the container with id aa9693fc408d88b1418c500204b083be0cb3b3aa72c035b613229141a403ed23 Oct 30 19:28:58.449778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.449742 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.449863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.449813 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.449863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.449845 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.574067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.574032 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:28:58.736211 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.736144 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" event={"ID":"8a404769-ddd9-473d-8d14-bd668ced40bd","Type":"ContainerStarted","Data":"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba"} Oct 30 19:28:58.736373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.736289 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" event={"ID":"8a404769-ddd9-473d-8d14-bd668ced40bd","Type":"ContainerStarted","Data":"aa9693fc408d88b1418c500204b083be0cb3b3aa72c035b613229141a403ed23"} Oct 30 19:28:58.761776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.761710 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podStartSLOduration=1.761689048 podStartE2EDuration="1.761689048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:58.759291245 +0000 UTC m=+544.735517190" watchObservedRunningTime="2026-10-30 19:28:58.761689048 +0000 UTC m=+544.737914994" Oct 30 19:28:58.872806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.872772 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:28:58.874125 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:28:58.874099 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62fcbd8_e151_4e4d_a525_8f08904e222c.slice/crio-9afd367540aac80017a45ff4ed085e9ac9a2506f2a6efc9a5137e560182ce133 WatchSource:0}: Error finding container 9afd367540aac80017a45ff4ed085e9ac9a2506f2a6efc9a5137e560182ce133: Status 404 returned error can't find the container with id 9afd367540aac80017a45ff4ed085e9ac9a2506f2a6efc9a5137e560182ce133 Oct 30 19:28:58.876722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.876689 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.876804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.876757 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:58.876804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:58.876788 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:28:59.152650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.152538 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:28:59.156207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.156177 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:28:59.156543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.156505 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:28:59.779092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.777436 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" event={"ID":"e62fcbd8-e151-4e4d-a525-8f08904e222c","Type":"ContainerStarted","Data":"8a29299f2d26051e3f02da3826ce2de8d43aace88b3023c12433c098a9fa1f08"} Oct 30 19:28:59.779092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.777544 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" event={"ID":"e62fcbd8-e151-4e4d-a525-8f08904e222c","Type":"ContainerStarted","Data":"9afd367540aac80017a45ff4ed085e9ac9a2506f2a6efc9a5137e560182ce133"} Oct 30 19:28:59.814532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:28:59.814468 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podStartSLOduration=1.81443256 podStartE2EDuration="1.81443256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:28:59.811496118 +0000 UTC m=+545.787722063" watchObservedRunningTime="2026-10-30 19:28:59.81443256 +0000 UTC m=+545.790658575" Oct 30 19:29:00.169358 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:00.169262 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.169967 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:00.169933 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:29:00.583470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:00.581504 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:29:00.583470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:00.581693 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:00.583470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:00.581741 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:01.151471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:01.148881 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.151471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:01.148948 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:29:01.575945 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:01.575905 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:01.576586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:01.576549 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:02.145926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:02.145884 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.146222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:02.146192 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:29:02.574947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:02.574908 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:02.575249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:02.575218 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:03.147467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:03.146909 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.147467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:03.146975 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:29:03.577069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:03.577008 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:03.577421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:03.577374 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:04.180498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:04.175900 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.180498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:04.175976 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 19:29:04.280230 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:04.280175 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:29:04.575380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:04.575340 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:04.575589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:04.575402 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:05.574909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:05.574866 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 19:29:05.575408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:05.574941 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 19:29:06.188602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.188564 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:29:06.188828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.188624 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:29:06.444571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.444477 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:29:06.444877 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.444848 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:29:06.446824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.446786 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:29:06.935147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.935117 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:29:06.935675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.935478 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:29:06.936322 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:06.936301 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:29:07.737400 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:29:07.737367 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:29:13.031799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.031761 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:29:13.035908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.035882 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.046963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.046920 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-639c15-openshift-default-dockercfg-vrz5l\"" Oct 30 19:29:13.050260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.050230 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:29:13.105057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.105009 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.105421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.105395 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt69r\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.105595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.105576 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.105716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.105702 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.105838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.105825 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.106042 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.106020 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.106181 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.106166 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.106287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.106272 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.106408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.106393 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207336 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207426 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt69r\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207532 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207561 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207628 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207688 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207714 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.207932 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.207765 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.208624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.208596 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.208733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.208624 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.208802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.208759 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.208993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.208967 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.209373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.209349 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.210863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.210837 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.214802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.214776 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.219055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.219035 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.220021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.219995 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt69r\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.221029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.221007 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.352466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.352369 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:13.636105 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.636075 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:29:13.637695 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:13.637667 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390ac4fa_5635_41aa_8817_52c771435684.slice/crio-b92ac3a044519e658ffeef1a39a31d8deabfe53d223110c62ba5045a2c522757 WatchSource:0}: Error finding container b92ac3a044519e658ffeef1a39a31d8deabfe53d223110c62ba5045a2c522757: Status 404 returned error can't find the container with id b92ac3a044519e658ffeef1a39a31d8deabfe53d223110c62ba5045a2c522757 Oct 30 19:29:13.640091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.640054 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:13.640176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.640124 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:13.640176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.640153 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:13.879420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.879377 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" event={"ID":"390ac4fa-5635-41aa-8817-52c771435684","Type":"ContainerStarted","Data":"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261"} Oct 30 19:29:13.879638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.879463 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" event={"ID":"390ac4fa-5635-41aa-8817-52c771435684","Type":"ContainerStarted","Data":"b92ac3a044519e658ffeef1a39a31d8deabfe53d223110c62ba5045a2c522757"} Oct 30 19:29:13.902153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:13.902043 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podStartSLOduration=1.9020282229999999 podStartE2EDuration="1.902028223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:13.900147864 +0000 UTC m=+559.876373811" watchObservedRunningTime="2026-10-30 19:29:13.902028223 +0000 UTC m=+559.878254166" Oct 30 19:29:14.352598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:14.352560 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:14.353847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:14.353819 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:14.353966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:14.353883 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:15.359579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:15.353960 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:15.359579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:15.354021 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:16.352926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:16.352881 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:16.353121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:16.352959 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:17.353467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:17.352872 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:17.353467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:17.352942 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:18.353035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:18.352988 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:18.353252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:18.353052 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:19.353045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:19.352999 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:19.353584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:19.353055 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:20.353691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:20.353650 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:29:20.354118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:20.353722 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 19:29:21.357802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:21.357769 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:21.358287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:21.358272 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:21.359131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:21.359110 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:29:24.400014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.399979 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v"] Oct 30 19:29:24.404984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.404965 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.413179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.413155 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:24.418049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.418026 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v"] Oct 30 19:29:24.424518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.424487 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.424648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.424546 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.424648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.424633 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsq72\" (UniqueName: \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.509679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.509645 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:29:24.553340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.553302 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.553540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.553352 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.553540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.553410 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsq72\" (UniqueName: \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.554546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.554513 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.559738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.559707 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.562994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.562952 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsq72\" (UniqueName: \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\") pod \"maas-api-revoke-sub-317ba8f4-gxf5v\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.721123 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.721089 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:24.999793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:24.999758 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v"] Oct 30 19:29:25.001562 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:25.001536 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ce8590_3148_4101_99f7_7bf3d0f1ac15.slice/crio-89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c WatchSource:0}: Error finding container 89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c: Status 404 returned error can't find the container with id 89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c Oct 30 19:29:25.048943 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:25.048908 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" event={"ID":"36ce8590-3148-4101-99f7-7bf3d0f1ac15","Type":"ContainerStarted","Data":"89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c"} Oct 30 19:29:27.884986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:27.884948 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv"] Oct 30 19:29:27.890370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:27.890352 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:27.898202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:27.898164 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv"] Oct 30 19:29:28.000472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.000388 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltpz\" (UniqueName: \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.000649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.000490 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.000754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.000722 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.072125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.072085 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" event={"ID":"36ce8590-3148-4101-99f7-7bf3d0f1ac15","Type":"ContainerStarted","Data":"f5c928246e8c4a723706e6e525ad8e2d50c64c87e7cb823a4123d276be670587"} Oct 30 19:29:28.102654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.102490 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltpz\" (UniqueName: \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.102654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.102567 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.102968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.102729 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.103738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.103708 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.105808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.105786 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.106836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.106787 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" podStartSLOduration=1.572612516 podStartE2EDuration="4.106770619s" totalImagesPullingTime="2.534158103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:28.102056037 +0000 UTC m=+574.078281982" watchObservedRunningTime="2026-10-30 19:29:28.106770619 +0000 UTC m=+574.082996564" Oct 30 19:29:28.117956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.117929 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltpz\" (UniqueName: \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\") pod \"maas-api-revoke-sub-ebd801da-zktlv\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.207565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.207527 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:28.299490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.298747 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6"] Oct 30 19:29:28.304205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.304175 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.317062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.317026 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6"] Oct 30 19:29:28.406163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.406065 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.406163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.406130 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.406403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.406306 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhtw\" (UniqueName: \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.483115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.483063 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:29:28.497284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.497262 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv"] Oct 30 19:29:28.499956 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:28.499923 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147168d7_4250_4442_9d13_b15fbfd2c936.slice/crio-54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854 WatchSource:0}: Error finding container 54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854: Status 404 returned error can't find the container with id 54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854 Oct 30 19:29:28.511346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.511315 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.511486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.511382 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.511552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.511510 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.511658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.511640 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhtw\" (UniqueName: \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.514656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.514633 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.520585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.520556 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhtw\" (UniqueName: \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\") pod \"maas-api-revoke-sub-2fe70ee8-f4gp6\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:28.628241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:28.628205 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:29.038134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.038104 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6"] Oct 30 19:29:29.040954 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:29.040923 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fb3dd2_2329_47a8_b148_e3d10d155e47.slice/crio-05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67 WatchSource:0}: Error finding container 05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67: Status 404 returned error can't find the container with id 05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67 Oct 30 19:29:29.080710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.080678 6907 generic.go:356] "Generic (PLEG): container finished" podID="36ce8590-3148-4101-99f7-7bf3d0f1ac15" containerID="f5c928246e8c4a723706e6e525ad8e2d50c64c87e7cb823a4123d276be670587" exitCode=0 Oct 30 19:29:29.080867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.080728 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" event={"ID":"36ce8590-3148-4101-99f7-7bf3d0f1ac15","Type":"ContainerDied","Data":"f5c928246e8c4a723706e6e525ad8e2d50c64c87e7cb823a4123d276be670587"} Oct 30 19:29:29.085628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.082748 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" event={"ID":"a5fb3dd2-2329-47a8-b148-e3d10d155e47","Type":"ContainerStarted","Data":"05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67"} Oct 30 19:29:29.089097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.089068 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" event={"ID":"147168d7-4250-4442-9d13-b15fbfd2c936","Type":"ContainerStarted","Data":"2dad88cc8c0d001f917240442ab8bea7cff7e7057f49c70e09b93d2d604d851e"} Oct 30 19:29:29.089227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.089117 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" event={"ID":"147168d7-4250-4442-9d13-b15fbfd2c936","Type":"ContainerStarted","Data":"54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854"} Oct 30 19:29:29.116887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:29.116822 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" podStartSLOduration=2.116800918 podStartE2EDuration="2.116800918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:29.1158682 +0000 UTC m=+575.092094172" watchObservedRunningTime="2026-10-30 19:29:29.116800918 +0000 UTC m=+575.093026863" Oct 30 19:29:30.120482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.118247 6907 generic.go:356] "Generic (PLEG): container finished" podID="147168d7-4250-4442-9d13-b15fbfd2c936" containerID="2dad88cc8c0d001f917240442ab8bea7cff7e7057f49c70e09b93d2d604d851e" exitCode=0 Oct 30 19:29:30.120482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.118562 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" event={"ID":"147168d7-4250-4442-9d13-b15fbfd2c936","Type":"ContainerDied","Data":"2dad88cc8c0d001f917240442ab8bea7cff7e7057f49c70e09b93d2d604d851e"} Oct 30 19:29:30.124511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.123599 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" event={"ID":"a5fb3dd2-2329-47a8-b148-e3d10d155e47","Type":"ContainerStarted","Data":"c88ba2fdab82033f0b14d2e763bccc6ac5770f38b6913374d7d3c39b8a1ad3de"} Oct 30 19:29:30.265520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.263277 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" podStartSLOduration=2.263250806 podStartE2EDuration="2.263250806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:30.199525591 +0000 UTC m=+576.175751537" watchObservedRunningTime="2026-10-30 19:29:30.263250806 +0000 UTC m=+576.239476751" Oct 30 19:29:30.489600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.489570 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:30.596551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.596518 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\") pod \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " Oct 30 19:29:30.596723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.596574 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\") pod \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " Oct 30 19:29:30.596723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.596672 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\" (UniqueName: \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\") pod \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\" (UID: \"36ce8590-3148-4101-99f7-7bf3d0f1ac15\") " Oct 30 19:29:30.596960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.596929 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca" pod "36ce8590-3148-4101-99f7-7bf3d0f1ac15" (UID: "36ce8590-3148-4101-99f7-7bf3d0f1ac15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:30.597282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.597266 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:30.598746 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.598715 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca" pod "36ce8590-3148-4101-99f7-7bf3d0f1ac15" (UID: "36ce8590-3148-4101-99f7-7bf3d0f1ac15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:30.598855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.598782 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72" pod "36ce8590-3148-4101-99f7-7bf3d0f1ac15" (UID: "36ce8590-3148-4101-99f7-7bf3d0f1ac15"). InnerVolumeSpecName "kube-api-access-bsq72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:30.698330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.698240 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ce8590-3148-4101-99f7-7bf3d0f1ac15-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:30.698330 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:30.698272 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsq72\" (UniqueName: \"kubernetes.io/projected/36ce8590-3148-4101-99f7-7bf3d0f1ac15-kube-api-access-bsq72\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:31.131188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.131158 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" Oct 30 19:29:31.131641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.131158 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v" event={"ID":"36ce8590-3148-4101-99f7-7bf3d0f1ac15","Type":"ContainerDied","Data":"89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c"} Oct 30 19:29:31.131641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.131334 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e545a1361312fdd440d9a418984cc084c128f40989572317a5c75085c03c4c" Oct 30 19:29:31.133160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.133135 6907 generic.go:356] "Generic (PLEG): container finished" podID="a5fb3dd2-2329-47a8-b148-e3d10d155e47" containerID="c88ba2fdab82033f0b14d2e763bccc6ac5770f38b6913374d7d3c39b8a1ad3de" exitCode=0 Oct 30 19:29:31.133298 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.133215 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" event={"ID":"a5fb3dd2-2329-47a8-b148-e3d10d155e47","Type":"ContainerDied","Data":"c88ba2fdab82033f0b14d2e763bccc6ac5770f38b6913374d7d3c39b8a1ad3de"} Oct 30 19:29:31.339379 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.339347 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:31.507948 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.507909 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\") pod \"147168d7-4250-4442-9d13-b15fbfd2c936\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " Oct 30 19:29:31.508126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.508046 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\") pod \"147168d7-4250-4442-9d13-b15fbfd2c936\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " Oct 30 19:29:31.508177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.508143 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\" (UniqueName: \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\") pod \"147168d7-4250-4442-9d13-b15fbfd2c936\" (UID: \"147168d7-4250-4442-9d13-b15fbfd2c936\") " Oct 30 19:29:31.508403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.508374 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca" pod "147168d7-4250-4442-9d13-b15fbfd2c936" (UID: "147168d7-4250-4442-9d13-b15fbfd2c936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:31.508630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.508611 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:31.510195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.510164 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca" pod "147168d7-4250-4442-9d13-b15fbfd2c936" (UID: "147168d7-4250-4442-9d13-b15fbfd2c936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:31.510300 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.510277 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz" pod "147168d7-4250-4442-9d13-b15fbfd2c936" (UID: "147168d7-4250-4442-9d13-b15fbfd2c936"). InnerVolumeSpecName "kube-api-access-5ltpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:31.610014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.609980 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/147168d7-4250-4442-9d13-b15fbfd2c936-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:31.610014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:31.610014 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltpz\" (UniqueName: \"kubernetes.io/projected/147168d7-4250-4442-9d13-b15fbfd2c936-kube-api-access-5ltpz\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:32.141200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.141166 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" Oct 30 19:29:32.141648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.141201 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv" event={"ID":"147168d7-4250-4442-9d13-b15fbfd2c936","Type":"ContainerDied","Data":"54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854"} Oct 30 19:29:32.141648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.141270 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54172dfbd47535c0590e2b1ee90b9acfcef047e06035726819db465cca998854" Oct 30 19:29:32.302375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.302352 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:32.420901 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.420812 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\") pod \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " Oct 30 19:29:32.420901 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.420869 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\") pod \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " Oct 30 19:29:32.421145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.420973 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\" (UniqueName: \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\") pod \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\" (UID: \"a5fb3dd2-2329-47a8-b148-e3d10d155e47\") " Oct 30 19:29:32.421285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.421260 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca" pod "a5fb3dd2-2329-47a8-b148-e3d10d155e47" (UID: "a5fb3dd2-2329-47a8-b148-e3d10d155e47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:32.421430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.421414 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:32.423154 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.423125 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw" pod "a5fb3dd2-2329-47a8-b148-e3d10d155e47" (UID: "a5fb3dd2-2329-47a8-b148-e3d10d155e47"). InnerVolumeSpecName "kube-api-access-nkhtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:32.423238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.423134 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca" pod "a5fb3dd2-2329-47a8-b148-e3d10d155e47" (UID: "a5fb3dd2-2329-47a8-b148-e3d10d155e47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:32.523094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.523061 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhtw\" (UniqueName: \"kubernetes.io/projected/a5fb3dd2-2329-47a8-b148-e3d10d155e47-kube-api-access-nkhtw\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:32.523094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:32.523090 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5fb3dd2-2329-47a8-b148-e3d10d155e47-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:33.148805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:33.148766 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" event={"ID":"a5fb3dd2-2329-47a8-b148-e3d10d155e47","Type":"ContainerDied","Data":"05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67"} Oct 30 19:29:33.148805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:33.148797 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6" Oct 30 19:29:33.149264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:33.148819 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05cab2b737479ef6a5cbc84db6eaf23eec3d55c80f94593f1fa8bb57c17cdd67" Oct 30 19:29:37.736408 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:29:37.736366 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:29:38.574969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.574928 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9"] Oct 30 19:29:38.576193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576167 6907 cpu_manager.go:414] "containerMap: removing container" podUID="36ce8590-3148-4101-99f7-7bf3d0f1ac15" containerName="revoke-keys" Oct 30 19:29:38.576193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576192 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ce8590-3148-4101-99f7-7bf3d0f1ac15" containerName="revoke-keys" Oct 30 19:29:38.576383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576257 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a5fb3dd2-2329-47a8-b148-e3d10d155e47" containerName="revoke-keys" Oct 30 19:29:38.576383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576265 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fb3dd2-2329-47a8-b148-e3d10d155e47" containerName="revoke-keys" Oct 30 19:29:38.576383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576286 6907 cpu_manager.go:414] "containerMap: removing container" podUID="147168d7-4250-4442-9d13-b15fbfd2c936" containerName="revoke-keys" Oct 30 19:29:38.576383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576294 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147168d7-4250-4442-9d13-b15fbfd2c936" containerName="revoke-keys" Oct 30 19:29:38.576661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576645 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fb3dd2-2329-47a8-b148-e3d10d155e47" containerName="revoke-keys" Oct 30 19:29:38.576722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576682 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ce8590-3148-4101-99f7-7bf3d0f1ac15" containerName="revoke-keys" Oct 30 19:29:38.576722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.576695 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="147168d7-4250-4442-9d13-b15fbfd2c936" containerName="revoke-keys" Oct 30 19:29:38.581713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.581408 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.586925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.586902 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:38.587216 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.587191 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:29:38.588635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.588611 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9"] Oct 30 19:29:38.634913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.634872 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwng\" (UniqueName: \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.635092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.634924 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.635134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.635084 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.736619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.736583 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.737003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.736709 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwng\" (UniqueName: \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.737003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.736744 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.737334 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.737316 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.738939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.738913 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.745948 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.745923 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwng\" (UniqueName: \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\") pod \"maas-api-revoke-sub-8060255e-rttj9\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:38.902103 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:38.902009 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:39.180025 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:39.179999 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9"] Oct 30 19:29:39.182164 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:39.182134 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c596a20_cd10_4925_bcc0_1fc3089bf53d.slice/crio-36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19 WatchSource:0}: Error finding container 36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19: Status 404 returned error can't find the container with id 36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19 Oct 30 19:29:40.055187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.055150 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz"] Oct 30 19:29:40.059458 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.059419 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.073509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.073480 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz"] Oct 30 19:29:40.153903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.153875 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb5n4\" (UniqueName: \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.154052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.154027 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.154095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.154062 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.195087 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.195051 6907 generic.go:356] "Generic (PLEG): container finished" podID="5c596a20-cd10-4925-bcc0-1fc3089bf53d" containerID="39237f8a9a608e6694261f67adb9deb2aa0248ce101932f80e0713fba5aa2db4" exitCode=0 Oct 30 19:29:40.195222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.195138 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" event={"ID":"5c596a20-cd10-4925-bcc0-1fc3089bf53d","Type":"ContainerDied","Data":"39237f8a9a608e6694261f67adb9deb2aa0248ce101932f80e0713fba5aa2db4"} Oct 30 19:29:40.195222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.195196 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" event={"ID":"5c596a20-cd10-4925-bcc0-1fc3089bf53d","Type":"ContainerStarted","Data":"36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19"} Oct 30 19:29:40.260543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.255337 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.260543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.255408 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.260543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.255574 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb5n4\" (UniqueName: \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.264489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.264399 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.283263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.283221 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.283492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.283224 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb5n4\" (UniqueName: \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\") pod \"maas-api-revoke-sub-023c8182-g97pz\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.377551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.377455 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:40.712649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:40.712457 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz"] Oct 30 19:29:41.203933 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.203898 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" event={"ID":"f359cbf3-3422-479d-a9a8-a7a95a17ea96","Type":"ContainerStarted","Data":"c689c82ebeab78eff1cdbbfc44c7225b52d2e7661996da3d6b82c8c341179d10"} Oct 30 19:29:41.204368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.203963 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" event={"ID":"f359cbf3-3422-479d-a9a8-a7a95a17ea96","Type":"ContainerStarted","Data":"4db3112c6be867d16acb46d571b4bf52b6da123b6d4b96de1763b25bc2253e79"} Oct 30 19:29:41.226505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.226438 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" podStartSLOduration=1.226423835 podStartE2EDuration="1.226423835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:41.223272721 +0000 UTC m=+587.199498668" watchObservedRunningTime="2026-10-30 19:29:41.226423835 +0000 UTC m=+587.202649778" Oct 30 19:29:41.359541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.359510 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:41.493416 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.493326 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\" (UniqueName: \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\") pod \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " Oct 30 19:29:41.493590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.493420 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\") pod \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " Oct 30 19:29:41.493648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.493591 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\") pod \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\" (UID: \"5c596a20-cd10-4925-bcc0-1fc3089bf53d\") " Oct 30 19:29:41.493932 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.493903 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca" pod "5c596a20-cd10-4925-bcc0-1fc3089bf53d" (UID: "5c596a20-cd10-4925-bcc0-1fc3089bf53d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:41.494135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.494112 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:41.495639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.495616 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca" pod "5c596a20-cd10-4925-bcc0-1fc3089bf53d" (UID: "5c596a20-cd10-4925-bcc0-1fc3089bf53d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:41.495639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.495628 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng" pod "5c596a20-cd10-4925-bcc0-1fc3089bf53d" (UID: "5c596a20-cd10-4925-bcc0-1fc3089bf53d"). InnerVolumeSpecName "kube-api-access-llwng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:41.596034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.596002 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c596a20-cd10-4925-bcc0-1fc3089bf53d-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:41.596034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:41.596033 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwng\" (UniqueName: \"kubernetes.io/projected/5c596a20-cd10-4925-bcc0-1fc3089bf53d-kube-api-access-llwng\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:42.211781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:42.211754 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" Oct 30 19:29:42.212256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:42.211798 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9" event={"ID":"5c596a20-cd10-4925-bcc0-1fc3089bf53d","Type":"ContainerDied","Data":"36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19"} Oct 30 19:29:42.212256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:42.211854 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36237504e34945b63560fd31b15d70db883e88a79527ac7d9815515e56e16a19" Oct 30 19:29:42.215841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:42.214051 6907 generic.go:356] "Generic (PLEG): container finished" podID="f359cbf3-3422-479d-a9a8-a7a95a17ea96" containerID="c689c82ebeab78eff1cdbbfc44c7225b52d2e7661996da3d6b82c8c341179d10" exitCode=0 Oct 30 19:29:42.215841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:42.214150 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" event={"ID":"f359cbf3-3422-479d-a9a8-a7a95a17ea96","Type":"ContainerDied","Data":"c689c82ebeab78eff1cdbbfc44c7225b52d2e7661996da3d6b82c8c341179d10"} Oct 30 19:29:43.378706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.378674 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:43.522777 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.522677 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\") pod \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " Oct 30 19:29:43.522954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.522870 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\") pod \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " Oct 30 19:29:43.522954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.522909 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\" (UniqueName: \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\") pod \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\" (UID: \"f359cbf3-3422-479d-a9a8-a7a95a17ea96\") " Oct 30 19:29:43.523299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.523260 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca" pod "f359cbf3-3422-479d-a9a8-a7a95a17ea96" (UID: "f359cbf3-3422-479d-a9a8-a7a95a17ea96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:43.523565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.523543 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:43.524992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.524964 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca" pod "f359cbf3-3422-479d-a9a8-a7a95a17ea96" (UID: "f359cbf3-3422-479d-a9a8-a7a95a17ea96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:43.525240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.525214 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4" pod "f359cbf3-3422-479d-a9a8-a7a95a17ea96" (UID: "f359cbf3-3422-479d-a9a8-a7a95a17ea96"). InnerVolumeSpecName "kube-api-access-bb5n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:43.624734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.624703 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb5n4\" (UniqueName: \"kubernetes.io/projected/f359cbf3-3422-479d-a9a8-a7a95a17ea96-kube-api-access-bb5n4\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:43.624734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:43.624730 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f359cbf3-3422-479d-a9a8-a7a95a17ea96-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:44.229238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:44.229203 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" Oct 30 19:29:44.229411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:44.229204 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz" event={"ID":"f359cbf3-3422-479d-a9a8-a7a95a17ea96","Type":"ContainerDied","Data":"4db3112c6be867d16acb46d571b4bf52b6da123b6d4b96de1763b25bc2253e79"} Oct 30 19:29:44.229411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:44.229369 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db3112c6be867d16acb46d571b4bf52b6da123b6d4b96de1763b25bc2253e79" Oct 30 19:29:47.527599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.527565 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44"] Oct 30 19:29:47.528194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528178 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5c596a20-cd10-4925-bcc0-1fc3089bf53d" containerName="revoke-keys" Oct 30 19:29:47.528251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528194 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c596a20-cd10-4925-bcc0-1fc3089bf53d" containerName="revoke-keys" Oct 30 19:29:47.528251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528208 6907 cpu_manager.go:414] "containerMap: removing container" podUID="f359cbf3-3422-479d-a9a8-a7a95a17ea96" containerName="revoke-keys" Oct 30 19:29:47.528251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528214 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f359cbf3-3422-479d-a9a8-a7a95a17ea96" containerName="revoke-keys" Oct 30 19:29:47.528396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528320 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="f359cbf3-3422-479d-a9a8-a7a95a17ea96" containerName="revoke-keys" Oct 30 19:29:47.528396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.528329 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c596a20-cd10-4925-bcc0-1fc3089bf53d" containerName="revoke-keys" Oct 30 19:29:47.532773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.532748 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.541021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.540992 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:47.541886 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.541122 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:29:47.542065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.542042 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44"] Oct 30 19:29:47.688459 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.688408 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.688652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.688611 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76b6\" (UniqueName: \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.688748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.688728 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.790050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.789962 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76b6\" (UniqueName: \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.790050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.790033 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.790248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.790112 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.790755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.790727 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.792408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.792384 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.804959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.804933 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76b6\" (UniqueName: \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\") pod \"maas-api-revoke-sub-880b704d-m7s44\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:47.858933 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:47.858900 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:48.160370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:48.160344 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44"] Oct 30 19:29:48.161576 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:48.161550 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177391a3_ec28_405c_817f_95aba15c9bde.slice/crio-8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee WatchSource:0}: Error finding container 8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee: Status 404 returned error can't find the container with id 8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee Oct 30 19:29:48.255681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:48.255645 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" event={"ID":"177391a3-ec28-405c-817f-95aba15c9bde","Type":"ContainerStarted","Data":"8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee"} Oct 30 19:29:49.296466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:49.293203 6907 generic.go:356] "Generic (PLEG): container finished" podID="177391a3-ec28-405c-817f-95aba15c9bde" containerID="e417c72171953aecaec01d7be5774a78e3bf0227ef30a7fc3e4fd399ee038820" exitCode=0 Oct 30 19:29:49.296466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:49.293375 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" event={"ID":"177391a3-ec28-405c-817f-95aba15c9bde","Type":"ContainerDied","Data":"e417c72171953aecaec01d7be5774a78e3bf0227ef30a7fc3e4fd399ee038820"} Oct 30 19:29:50.454889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.454858 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:50.542324 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.542291 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\" (UniqueName: \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\") pod \"177391a3-ec28-405c-817f-95aba15c9bde\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " Oct 30 19:29:50.542522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.542356 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\") pod \"177391a3-ec28-405c-817f-95aba15c9bde\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " Oct 30 19:29:50.542522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.542424 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\") pod \"177391a3-ec28-405c-817f-95aba15c9bde\" (UID: \"177391a3-ec28-405c-817f-95aba15c9bde\") " Oct 30 19:29:50.542780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.542754 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca" pod "177391a3-ec28-405c-817f-95aba15c9bde" (UID: "177391a3-ec28-405c-817f-95aba15c9bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:50.542957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.542940 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:50.544573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.544546 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca" pod "177391a3-ec28-405c-817f-95aba15c9bde" (UID: "177391a3-ec28-405c-817f-95aba15c9bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:50.544662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.544613 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6" pod "177391a3-ec28-405c-817f-95aba15c9bde" (UID: "177391a3-ec28-405c-817f-95aba15c9bde"). InnerVolumeSpecName "kube-api-access-b76b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:50.647465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.647409 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177391a3-ec28-405c-817f-95aba15c9bde-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:50.647784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:50.647759 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76b6\" (UniqueName: \"kubernetes.io/projected/177391a3-ec28-405c-817f-95aba15c9bde-kube-api-access-b76b6\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:51.310631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:51.310592 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" Oct 30 19:29:51.310631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:51.310608 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44" event={"ID":"177391a3-ec28-405c-817f-95aba15c9bde","Type":"ContainerDied","Data":"8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee"} Oct 30 19:29:51.310840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:51.310671 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e1eff9ea7c3cd730a5829ca59fa2d4766c099e824115908cc2e893232375dee" Oct 30 19:29:52.792143 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.792103 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq"] Oct 30 19:29:52.793790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.793763 6907 cpu_manager.go:414] "containerMap: removing container" podUID="177391a3-ec28-405c-817f-95aba15c9bde" containerName="revoke-keys" Oct 30 19:29:52.793944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.793927 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177391a3-ec28-405c-817f-95aba15c9bde" containerName="revoke-keys" Oct 30 19:29:52.794285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.794263 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="177391a3-ec28-405c-817f-95aba15c9bde" containerName="revoke-keys" Oct 30 19:29:52.812254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.812177 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:52.827637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.827428 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:29:52.827637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.827602 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:29:52.838493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.832176 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq"] Oct 30 19:29:52.989960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.989926 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:52.990135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.990068 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdt2r\" (UniqueName: \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:52.990184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:52.990145 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.091498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.091387 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdt2r\" (UniqueName: \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.091638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.091509 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.091638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.091597 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.092291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.092264 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.093946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.093924 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.102951 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.102918 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdt2r\" (UniqueName: \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\") pod \"maas-api-revoke-sub-d6bc7d0c-vn4pq\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.205921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.205880 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:53.486981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:53.486948 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq"] Oct 30 19:29:53.488143 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:53.488116 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a3049e_e477_40e2_9c10_99d258d6ae99.slice/crio-cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690 WatchSource:0}: Error finding container cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690: Status 404 returned error can't find the container with id cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690 Oct 30 19:29:54.228308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.228282 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:29:54.230555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.230531 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:29:54.251780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.251755 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:29:54.254214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.254194 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:29:54.335421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.335387 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" event={"ID":"37a3049e-e477-40e2-9c10-99d258d6ae99","Type":"ContainerStarted","Data":"f49c970ef02f73f2dc552098bae8dce9b799e4a16e24c02f97446a94d141fc95"} Oct 30 19:29:54.335611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.335453 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" event={"ID":"37a3049e-e477-40e2-9c10-99d258d6ae99","Type":"ContainerStarted","Data":"cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690"} Oct 30 19:29:54.354169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.354104 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" podStartSLOduration=2.354087408 podStartE2EDuration="2.354087408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:54.353326801 +0000 UTC m=+600.329552760" watchObservedRunningTime="2026-10-30 19:29:54.354087408 +0000 UTC m=+600.330313353" Oct 30 19:29:54.402593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.402561 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8"] Oct 30 19:29:54.407010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.406991 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.417481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.414912 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8"] Oct 30 19:29:54.529888 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.526270 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhb5\" (UniqueName: \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.529888 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.526350 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.529888 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.526526 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.640469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.638692 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.640469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.638890 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhb5\" (UniqueName: \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.640469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.638943 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.644094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.641485 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.649142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.647657 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.650520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.650493 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhb5\" (UniqueName: \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\") pod \"maas-api-revoke-sub-1225c727-sbvc8\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:54.722490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:54.722457 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:55.005536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:55.005502 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8"] Oct 30 19:29:55.006544 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:29:55.006517 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721819bb_d6d8_4357_9fbd_e868d8343dd5.slice/crio-cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0 WatchSource:0}: Error finding container cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0: Status 404 returned error can't find the container with id cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0 Oct 30 19:29:55.345918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:55.345837 6907 generic.go:356] "Generic (PLEG): container finished" podID="37a3049e-e477-40e2-9c10-99d258d6ae99" containerID="f49c970ef02f73f2dc552098bae8dce9b799e4a16e24c02f97446a94d141fc95" exitCode=0 Oct 30 19:29:55.345918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:55.345881 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" event={"ID":"37a3049e-e477-40e2-9c10-99d258d6ae99","Type":"ContainerDied","Data":"f49c970ef02f73f2dc552098bae8dce9b799e4a16e24c02f97446a94d141fc95"} Oct 30 19:29:55.347502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:55.347474 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" event={"ID":"721819bb-d6d8-4357-9fbd-e868d8343dd5","Type":"ContainerStarted","Data":"cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0"} Oct 30 19:29:56.401798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.401063 6907 generic.go:356] "Generic (PLEG): container finished" podID="721819bb-d6d8-4357-9fbd-e868d8343dd5" containerID="11e184fae3a05457f9c60f383476680ac4dde7602153a4e521b3ba28f030a0d5" exitCode=0 Oct 30 19:29:56.401798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.401594 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" event={"ID":"721819bb-d6d8-4357-9fbd-e868d8343dd5","Type":"ContainerDied","Data":"11e184fae3a05457f9c60f383476680ac4dde7602153a4e521b3ba28f030a0d5"} Oct 30 19:29:56.561084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.561058 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:56.571038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.571015 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\" (UniqueName: \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\") pod \"37a3049e-e477-40e2-9c10-99d258d6ae99\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " Oct 30 19:29:56.571157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.571132 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\") pod \"37a3049e-e477-40e2-9c10-99d258d6ae99\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " Oct 30 19:29:56.571192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.571156 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\") pod \"37a3049e-e477-40e2-9c10-99d258d6ae99\" (UID: \"37a3049e-e477-40e2-9c10-99d258d6ae99\") " Oct 30 19:29:56.571498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.571468 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca" pod "37a3049e-e477-40e2-9c10-99d258d6ae99" (UID: "37a3049e-e477-40e2-9c10-99d258d6ae99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:56.573121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.573098 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r" pod "37a3049e-e477-40e2-9c10-99d258d6ae99" (UID: "37a3049e-e477-40e2-9c10-99d258d6ae99"). InnerVolumeSpecName "kube-api-access-qdt2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:56.573201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.573138 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca" pod "37a3049e-e477-40e2-9c10-99d258d6ae99" (UID: "37a3049e-e477-40e2-9c10-99d258d6ae99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:56.672228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.672137 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:56.672228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.672171 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37a3049e-e477-40e2-9c10-99d258d6ae99-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:56.672228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:56.672181 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdt2r\" (UniqueName: \"kubernetes.io/projected/37a3049e-e477-40e2-9c10-99d258d6ae99-kube-api-access-qdt2r\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:57.183046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.182995 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:29:57.410993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.410961 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" Oct 30 19:29:57.410993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.410981 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq" event={"ID":"37a3049e-e477-40e2-9c10-99d258d6ae99","Type":"ContainerDied","Data":"cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690"} Oct 30 19:29:57.411455 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.411037 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd497e2e2752860b472cc8be6bc132fc10cbe5060eee7e3119b03811e1db3690" Oct 30 19:29:57.566261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.566233 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:57.582727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.582702 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\") pod \"721819bb-d6d8-4357-9fbd-e868d8343dd5\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " Oct 30 19:29:57.582843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.582799 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\") pod \"721819bb-d6d8-4357-9fbd-e868d8343dd5\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " Oct 30 19:29:57.582911 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.582898 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\" (UniqueName: \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\") pod \"721819bb-d6d8-4357-9fbd-e868d8343dd5\" (UID: \"721819bb-d6d8-4357-9fbd-e868d8343dd5\") " Oct 30 19:29:57.583043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.583020 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca" pod "721819bb-d6d8-4357-9fbd-e868d8343dd5" (UID: "721819bb-d6d8-4357-9fbd-e868d8343dd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:29:57.583328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.583309 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:57.584874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.584838 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca" pod "721819bb-d6d8-4357-9fbd-e868d8343dd5" (UID: "721819bb-d6d8-4357-9fbd-e868d8343dd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:29:57.584970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.584954 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5" pod "721819bb-d6d8-4357-9fbd-e868d8343dd5" (UID: "721819bb-d6d8-4357-9fbd-e868d8343dd5"). InnerVolumeSpecName "kube-api-access-zkhb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:29:57.685057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.685019 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkhb5\" (UniqueName: \"kubernetes.io/projected/721819bb-d6d8-4357-9fbd-e868d8343dd5-kube-api-access-zkhb5\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:57.685057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:57.685052 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/721819bb-d6d8-4357-9fbd-e868d8343dd5-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.488504 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514285 6907 cpu_manager.go:414] "containerMap: removing container" podUID="721819bb-d6d8-4357-9fbd-e868d8343dd5" containerName="revoke-keys" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514314 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721819bb-d6d8-4357-9fbd-e868d8343dd5" containerName="revoke-keys" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514358 6907 cpu_manager.go:414] "containerMap: removing container" podUID="37a3049e-e477-40e2-9c10-99d258d6ae99" containerName="revoke-keys" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514366 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a3049e-e477-40e2-9c10-99d258d6ae99" containerName="revoke-keys" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514588 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="721819bb-d6d8-4357-9fbd-e868d8343dd5" containerName="revoke-keys" Oct 30 19:29:58.611524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.514609 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a3049e-e477-40e2-9c10-99d258d6ae99" containerName="revoke-keys" Oct 30 19:29:58.662469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.626639 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:29:58.662469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.626775 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.709529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.690535 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" event={"ID":"721819bb-d6d8-4357-9fbd-e868d8343dd5","Type":"ContainerDied","Data":"cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0"} Oct 30 19:29:58.709529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.690625 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6f06badf4dd6dafef7a553f44b81859d1b03eeb159f9db14573c421b3d5ce0" Oct 30 19:29:58.709529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.690724 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.728674 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.770847 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.770941 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.770997 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.771032 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.771147 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hhs9\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.771199 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.771225 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.771436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.771329 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.878494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.878434 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.878751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.878725 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.878888 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.878871 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879038 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hhs9\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879209 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879323 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879497 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879754 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.879991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.879972 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.887663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.887631 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.888065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.888041 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.896473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.888510 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.896473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.888765 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.896818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.896787 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.901689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.901662 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.978705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.978601 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:58.994490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:58.980907 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hhs9\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:59.025483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.018476 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:59.155009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.154700 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:29:59.936468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.928270 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:29:59.970342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.970297 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:59.974665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.974628 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:29:59.974880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:29:59.974856 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:00.423468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.374352 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:00.637522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.635807 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:00.643517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.638616 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:00.657374 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.657300 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:30:00.659029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.658997 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:00.659743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.659714 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:00.730894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.730850 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:00.739828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.739785 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:00.740266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.740233 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhfp7\" (UniqueName: \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:00.761734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.729743 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:00.762011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.761993 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:00.762175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.762161 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:00.762408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.762393 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:00.812491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.795267 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:00.812491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.795394 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:00.858992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.858953 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56p2q\" (UniqueName: \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:00.859732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.859706 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:00.865469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.861546 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:00.866364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.866336 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:00.884504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.884471 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wgr\" (UniqueName: \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:00.907952 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.906297 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" event={"ID":"04e9ff52-ea3d-4372-b16b-7e6739738397","Type":"ContainerStarted","Data":"2afc535e158f631c56ac25a92fd5169ae137977999e497617420f282b029c61d"} Oct 30 19:30:00.933935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.933887 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" event={"ID":"04e9ff52-ea3d-4372-b16b-7e6739738397","Type":"ContainerStarted","Data":"372a534927dc32b876b9f8f078366e1a2e56486d055e417404ea456f86a723d3"} Oct 30 19:30:00.934326 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.907907 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhfp7\" (UniqueName: \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:00.935794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:00.935768 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:01.036475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.029180 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podStartSLOduration=3.029155801 podStartE2EDuration="3.029155801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:00.932002318 +0000 UTC m=+606.908228264" watchObservedRunningTime="2026-10-30 19:30:01.029155801 +0000 UTC m=+607.005381745" Oct 30 19:30:01.067470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.042187 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhfp7\" (UniqueName: \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\") pod \"maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:01.068166 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.042694 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvd5\" (UniqueName: \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.068420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.068393 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wgr\" (UniqueName: \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:01.069062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.069039 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56p2q\" (UniqueName: \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:01.070478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.070458 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:01.071700 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.071680 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:01.083479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.071545 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:01.083726 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.072867 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:01.083939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.083896 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.091614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.091582 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56p2q\" (UniqueName: \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\") pod \"maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:01.124513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.116571 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wgr\" (UniqueName: \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:01.163267 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.162103 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:30:01.163566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.163537 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:01.163758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.163731 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:01.180053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.179920 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:01.180053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.179984 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.188964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.188928 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.207697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.207662 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvd5\" (UniqueName: \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.208954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.190864 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:01.209345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.190731 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.262006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.261963 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvd5\" (UniqueName: \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\") pod \"maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.262489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.262464 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:01.340486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.337654 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:01.383392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.383345 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:01.395208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.395164 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:01.395392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.395232 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.520477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.518266 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:01.641997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.641623 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:01.641997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:01.641691 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:02.181798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.181303 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:02.181798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.181359 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:02.324671 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:02.324636 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a70d09c_2dc3_4973_bd8c_89c971268f1c.slice/crio-32fe9981d56a49ecbba30d839f33e8aea141d98df405b754779ac9caa633267f WatchSource:0}: Error finding container 32fe9981d56a49ecbba30d839f33e8aea141d98df405b754779ac9caa633267f: Status 404 returned error can't find the container with id 32fe9981d56a49ecbba30d839f33e8aea141d98df405b754779ac9caa633267f Oct 30 19:30:02.357480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.353503 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:02.373492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.372637 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:02.399627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.399572 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:02.614149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.590774 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn"] Oct 30 19:30:02.618626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.617811 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn"] Oct 30 19:30:02.618626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.617961 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.622129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.622082 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:02.746798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.746756 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:02.808225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.808194 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cb5x\" (UniqueName: \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.808348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.808307 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.808389 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.808348 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.909086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.909047 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cb5x\" (UniqueName: \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.909281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.909175 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.909281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.909218 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.909921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.909900 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.911559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.911534 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.917753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.917729 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cb5x\" (UniqueName: \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\") pod \"maas-api-revoke-sub-1e916cf6-9wqtn\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:02.974114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.974078 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerStarted","Data":"9efd557b7ecd796bb61d89beedc5d6b1ad1497fc30f69d38d57179f20c227b9c"} Oct 30 19:30:02.976228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.976195 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerStarted","Data":"163fd778209a4bb68deb651aa6f4384726a37c7f30787120b432a71a47ccbeef"} Oct 30 19:30:02.977737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.977711 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerStarted","Data":"92e0ae219a362c94bb0281d5ce5e7ef4705242c93fffa33fe685edd378b2e43e"} Oct 30 19:30:02.979511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.979484 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerStarted","Data":"32fe9981d56a49ecbba30d839f33e8aea141d98df405b754779ac9caa633267f"} Oct 30 19:30:02.996366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:02.996318 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" podStartSLOduration=2.996302654 podStartE2EDuration="2.996302654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:02.995154345 +0000 UTC m=+608.971380291" watchObservedRunningTime="2026-10-30 19:30:02.996302654 +0000 UTC m=+608.972528644" Oct 30 19:30:03.009761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:03.009735 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:03.157972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:03.157512 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:03.157972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:03.157568 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:03.322083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:03.322053 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn"] Oct 30 19:30:03.323522 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:03.323492 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30917b9a_0bc0_4727_b757_812f64ff420b.slice/crio-acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb WatchSource:0}: Error finding container acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb: Status 404 returned error can't find the container with id acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb Oct 30 19:30:04.010996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.009044 6907 generic.go:356] "Generic (PLEG): container finished" podID="ce971f83-6566-4031-840a-3927065301c7" containerID="63cc3e25db3c9a62c26e48517d95660a8fb017c9e670ecc1267d6cd652abde2b" exitCode=60 Oct 30 19:30:04.010996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.009193 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerDied","Data":"63cc3e25db3c9a62c26e48517d95660a8fb017c9e670ecc1267d6cd652abde2b"} Oct 30 19:30:04.010996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.009683 6907 scope.go:122] "RemoveContainer" containerID="63cc3e25db3c9a62c26e48517d95660a8fb017c9e670ecc1267d6cd652abde2b" Oct 30 19:30:04.015119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.014999 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" event={"ID":"30917b9a-0bc0-4727-b757-812f64ff420b","Type":"ContainerStarted","Data":"f260c1dece56535675a49f15d786c9fccbbc2225d867ac5afc55c01d2890dfc7"} Oct 30 19:30:04.015119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.015064 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" event={"ID":"30917b9a-0bc0-4727-b757-812f64ff420b","Type":"ContainerStarted","Data":"acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb"} Oct 30 19:30:04.020592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.020567 6907 generic.go:356] "Generic (PLEG): container finished" podID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerID="4d0d337dfe39e4c0d480f8a761dabf6be5110856adaa0519a20d55ecd6fdd545" exitCode=60 Oct 30 19:30:04.020729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.020707 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerDied","Data":"4d0d337dfe39e4c0d480f8a761dabf6be5110856adaa0519a20d55ecd6fdd545"} Oct 30 19:30:04.021269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.020923 6907 scope.go:122] "RemoveContainer" containerID="4d0d337dfe39e4c0d480f8a761dabf6be5110856adaa0519a20d55ecd6fdd545" Oct 30 19:30:04.023035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.023014 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerStarted","Data":"290c34861619a69f221e6410ff5178a87787f86564b76247bc455e6b08230353"} Oct 30 19:30:04.023282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.023266 6907 scope.go:122] "RemoveContainer" containerID="290c34861619a69f221e6410ff5178a87787f86564b76247bc455e6b08230353" Oct 30 19:30:04.024851 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.024830 6907 generic.go:356] "Generic (PLEG): container finished" podID="5ad8080f-962e-4844-8043-e0673635bd87" containerID="1fb8ffd18fbdcb35eef7e078a15f3d2cf7d0b19299cd7cf274883ff0bea095c9" exitCode=60 Oct 30 19:30:04.024923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.024871 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerDied","Data":"1fb8ffd18fbdcb35eef7e078a15f3d2cf7d0b19299cd7cf274883ff0bea095c9"} Oct 30 19:30:04.025340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.025137 6907 scope.go:122] "RemoveContainer" containerID="1fb8ffd18fbdcb35eef7e078a15f3d2cf7d0b19299cd7cf274883ff0bea095c9" Oct 30 19:30:04.093511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.093458 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" podStartSLOduration=2.093428798 podStartE2EDuration="2.093428798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:04.092213077 +0000 UTC m=+610.068439022" watchObservedRunningTime="2026-10-30 19:30:04.093428798 +0000 UTC m=+610.069654748" Oct 30 19:30:04.157935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.157902 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:04.158225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:04.158194 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:05.057745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.057707 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerStarted","Data":"c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc"} Oct 30 19:30:05.059862 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.059830 6907 generic.go:356] "Generic (PLEG): container finished" podID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerID="290c34861619a69f221e6410ff5178a87787f86564b76247bc455e6b08230353" exitCode=60 Oct 30 19:30:05.059989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.059911 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerDied","Data":"290c34861619a69f221e6410ff5178a87787f86564b76247bc455e6b08230353"} Oct 30 19:30:05.059989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.059968 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerStarted","Data":"dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad"} Oct 30 19:30:05.062093 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.062065 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerStarted","Data":"bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3"} Oct 30 19:30:05.064114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.064086 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerStarted","Data":"5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1"} Oct 30 19:30:05.065741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.065721 6907 generic.go:356] "Generic (PLEG): container finished" podID="30917b9a-0bc0-4727-b757-812f64ff420b" containerID="f260c1dece56535675a49f15d786c9fccbbc2225d867ac5afc55c01d2890dfc7" exitCode=0 Oct 30 19:30:05.065831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.065746 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" event={"ID":"30917b9a-0bc0-4727-b757-812f64ff420b","Type":"ContainerDied","Data":"f260c1dece56535675a49f15d786c9fccbbc2225d867ac5afc55c01d2890dfc7"} Oct 30 19:30:05.122204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.122146 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" podStartSLOduration=5.122127822 podStartE2EDuration="5.122127822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.118946954 +0000 UTC m=+611.095172898" watchObservedRunningTime="2026-10-30 19:30:05.122127822 +0000 UTC m=+611.098353769" Oct 30 19:30:05.156265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.155857 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:05.156265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.155918 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:05.159209 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.159142 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" podStartSLOduration=5.159128477 podStartE2EDuration="5.159128477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.146678865 +0000 UTC m=+611.122904809" watchObservedRunningTime="2026-10-30 19:30:05.159128477 +0000 UTC m=+611.135354424" Oct 30 19:30:05.172480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:05.172416 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" podStartSLOduration=5.172401744 podStartE2EDuration="5.172401744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:05.169033551 +0000 UTC m=+611.145259497" watchObservedRunningTime="2026-10-30 19:30:05.172401744 +0000 UTC m=+611.148627714" Oct 30 19:30:06.075961 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.075917 6907 generic.go:356] "Generic (PLEG): container finished" podID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerID="c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc" exitCode=60 Oct 30 19:30:06.076428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.076005 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerDied","Data":"c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc"} Oct 30 19:30:06.076428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.076124 6907 scope.go:122] "RemoveContainer" containerID="4d0d337dfe39e4c0d480f8a761dabf6be5110856adaa0519a20d55ecd6fdd545" Oct 30 19:30:06.076428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.076283 6907 scope.go:122] "RemoveContainer" containerID="c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc" Oct 30 19:30:06.076609 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:06.076568 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg_odh-ai-gateway-infra(3a70d09c-2dc3-4973-bd8c-89c971268f1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" Oct 30 19:30:06.078135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.078115 6907 generic.go:356] "Generic (PLEG): container finished" podID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerID="dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad" exitCode=60 Oct 30 19:30:06.078240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.078190 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerDied","Data":"dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad"} Oct 30 19:30:06.078496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.078480 6907 scope.go:122] "RemoveContainer" containerID="dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad" Oct 30 19:30:06.078764 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:06.078738 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt_odh-ai-gateway-infra(85ce9191-a5cc-4845-81b0-5a93423295e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" Oct 30 19:30:06.080143 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.080119 6907 generic.go:356] "Generic (PLEG): container finished" podID="5ad8080f-962e-4844-8043-e0673635bd87" containerID="bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3" exitCode=60 Oct 30 19:30:06.080250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.080201 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerDied","Data":"bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3"} Oct 30 19:30:06.080396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.080375 6907 scope.go:122] "RemoveContainer" containerID="bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3" Oct 30 19:30:06.080645 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:06.080621 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974_odh-ai-gateway-infra(5ad8080f-962e-4844-8043-e0673635bd87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" podUID="5ad8080f-962e-4844-8043-e0673635bd87" Oct 30 19:30:06.082433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.082409 6907 generic.go:356] "Generic (PLEG): container finished" podID="ce971f83-6566-4031-840a-3927065301c7" containerID="5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1" exitCode=60 Oct 30 19:30:06.082547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.082527 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerDied","Data":"5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1"} Oct 30 19:30:06.082657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.082643 6907 scope.go:122] "RemoveContainer" containerID="5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1" Oct 30 19:30:06.082910 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:06.082866 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp_odh-ai-gateway-infra(ce971f83-6566-4031-840a-3927065301c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" podUID="ce971f83-6566-4031-840a-3927065301c7" Oct 30 19:30:06.090515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.090498 6907 scope.go:122] "RemoveContainer" containerID="290c34861619a69f221e6410ff5178a87787f86564b76247bc455e6b08230353" Oct 30 19:30:06.105976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.105953 6907 scope.go:122] "RemoveContainer" containerID="1fb8ffd18fbdcb35eef7e078a15f3d2cf7d0b19299cd7cf274883ff0bea095c9" Oct 30 19:30:06.121012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.120981 6907 scope.go:122] "RemoveContainer" containerID="63cc3e25db3c9a62c26e48517d95660a8fb017c9e670ecc1267d6cd652abde2b" Oct 30 19:30:06.194242 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.180880 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:06.194242 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.180942 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:06.393762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.393727 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:06.476320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.476281 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\" (UniqueName: \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\") pod \"30917b9a-0bc0-4727-b757-812f64ff420b\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " Oct 30 19:30:06.476540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.476389 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\") pod \"30917b9a-0bc0-4727-b757-812f64ff420b\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " Oct 30 19:30:06.476629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.476605 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\") pod \"30917b9a-0bc0-4727-b757-812f64ff420b\" (UID: \"30917b9a-0bc0-4727-b757-812f64ff420b\") " Oct 30 19:30:06.476759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.476730 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca" pod "30917b9a-0bc0-4727-b757-812f64ff420b" (UID: "30917b9a-0bc0-4727-b757-812f64ff420b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:06.477045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.477027 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:06.478539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.478512 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x" pod "30917b9a-0bc0-4727-b757-812f64ff420b" (UID: "30917b9a-0bc0-4727-b757-812f64ff420b"). InnerVolumeSpecName "kube-api-access-7cb5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:06.478721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.478705 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca" pod "30917b9a-0bc0-4727-b757-812f64ff420b" (UID: "30917b9a-0bc0-4727-b757-812f64ff420b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:06.578696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.578666 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30917b9a-0bc0-4727-b757-812f64ff420b-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:06.578696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.578693 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cb5x\" (UniqueName: \"kubernetes.io/projected/30917b9a-0bc0-4727-b757-812f64ff420b-kube-api-access-7cb5x\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:06.680013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.679931 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:30:06.681010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.680982 6907 cpu_manager.go:414] "containerMap: removing container" podUID="30917b9a-0bc0-4727-b757-812f64ff420b" containerName="revoke-keys" Oct 30 19:30:06.681010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.681010 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30917b9a-0bc0-4727-b757-812f64ff420b" containerName="revoke-keys" Oct 30 19:30:06.681222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.681206 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="30917b9a-0bc0-4727-b757-812f64ff420b" containerName="revoke-keys" Oct 30 19:30:06.685688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.685664 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.689996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.689971 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6cf459\"/\"kube-root-ca.crt\"" Oct 30 19:30:06.690403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.690384 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6cf459\"/\"default-dockercfg-jsz2z\"" Oct 30 19:30:06.690586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.690499 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6cf459\"/\"openshift-service-ca.crt\"" Oct 30 19:30:06.690676 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.690602 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6cf459\"/\"distinct-w1-6cf459-kserve-self-signed-certs\"" Oct 30 19:30:06.695086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.695060 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:30:06.781185 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781142 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.781185 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781184 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.781398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781311 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.781545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781525 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.781608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781569 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.781608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.781600 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gfd\" (UniqueName: \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884207 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884288 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gfd\" (UniqueName: \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884335 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884364 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884457 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.884663 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.885529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.885364 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.886179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.885658 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.886633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.886596 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.932571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.926117 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.932571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.926466 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:06.946416 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:06.944072 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gfd\" (UniqueName: \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\") pod \"distinct-w1-6cf459-kserve-59f664d5d7-rqbmt\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:07.005702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.005394 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:07.103476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.102726 6907 scope.go:122] "RemoveContainer" containerID="dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad" Oct 30 19:30:07.103476 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:07.103025 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt_odh-ai-gateway-infra(85ce9191-a5cc-4845-81b0-5a93423295e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" Oct 30 19:30:07.108268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.108227 6907 scope.go:122] "RemoveContainer" containerID="bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3" Oct 30 19:30:07.108681 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:07.108640 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974_odh-ai-gateway-infra(5ad8080f-962e-4844-8043-e0673635bd87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" podUID="5ad8080f-962e-4844-8043-e0673635bd87" Oct 30 19:30:07.111015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.110989 6907 scope.go:122] "RemoveContainer" containerID="5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1" Oct 30 19:30:07.111323 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:07.111299 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp_odh-ai-gateway-infra(ce971f83-6566-4031-840a-3927065301c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" podUID="ce971f83-6566-4031-840a-3927065301c7" Oct 30 19:30:07.113687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.113665 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" Oct 30 19:30:07.113827 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.113751 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn" event={"ID":"30917b9a-0bc0-4727-b757-812f64ff420b","Type":"ContainerDied","Data":"acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb"} Oct 30 19:30:07.113827 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.113801 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acb8f0cb838cc002b7f754d0e2e49845720755b8103c8a7879147addcf0b71cb" Oct 30 19:30:07.119172 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.119150 6907 scope.go:122] "RemoveContainer" containerID="c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc" Oct 30 19:30:07.121211 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:07.120196 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg_odh-ai-gateway-infra(3a70d09c-2dc3-4973-bd8c-89c971268f1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" Oct 30 19:30:07.166670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.159718 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 19:30:07.166670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.159780 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 19:30:07.369920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.369893 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:30:07.371496 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:07.371435 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c9ffb0_c8ec_4663_9880_745dabf09139.slice/crio-66a74f500a3620fb88bc461e8b3c4e42df29b26eb599cd3d47ba03e28f371e1b WatchSource:0}: Error finding container 66a74f500a3620fb88bc461e8b3c4e42df29b26eb599cd3d47ba03e28f371e1b: Status 404 returned error can't find the container with id 66a74f500a3620fb88bc461e8b3c4e42df29b26eb599cd3d47ba03e28f371e1b Oct 30 19:30:07.473729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.468807 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px"] Oct 30 19:30:07.479503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.476420 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px"] Oct 30 19:30:07.479503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.476563 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.481141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.481117 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:07.628343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.628253 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.628343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.628309 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.628552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.628500 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx527\" (UniqueName: \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.730054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.729270 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.730054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.729373 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx527\" (UniqueName: \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.730054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.729588 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.730369 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.730079 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.732471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.732416 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.737573 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:07.737538 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:30:07.745218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.745185 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx527\" (UniqueName: \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\") pod \"maas-api-revoke-sub-7b41ff9f-rh2px\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:07.819404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:07.819369 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:08.102506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.102469 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px"] Oct 30 19:30:08.103593 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:08.103562 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4cdf11_f521_403d_9465_69c59fcb1418.slice/crio-0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581 WatchSource:0}: Error finding container 0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581: Status 404 returned error can't find the container with id 0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581 Oct 30 19:30:08.129930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.129890 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" event={"ID":"b0c9ffb0-c8ec-4663-9880-745dabf09139","Type":"ContainerStarted","Data":"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7"} Oct 30 19:30:08.130050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.129953 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" event={"ID":"b0c9ffb0-c8ec-4663-9880-745dabf09139","Type":"ContainerStarted","Data":"66a74f500a3620fb88bc461e8b3c4e42df29b26eb599cd3d47ba03e28f371e1b"} Oct 30 19:30:08.132476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.132396 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" event={"ID":"7d4cdf11-f521-403d-9465-69c59fcb1418","Type":"ContainerStarted","Data":"0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581"} Oct 30 19:30:08.520858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.520818 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:30:08.522887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.522484 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:30:08.525008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.524981 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:30:08.546162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:08.546110 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podStartSLOduration=2.546096466 podStartE2EDuration="2.546096466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:08.151680114 +0000 UTC m=+614.127906058" watchObservedRunningTime="2026-10-30 19:30:08.546096466 +0000 UTC m=+614.522322410" Oct 30 19:30:09.006343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:09.006295 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:09.144047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:09.144011 6907 generic.go:356] "Generic (PLEG): container finished" podID="7d4cdf11-f521-403d-9465-69c59fcb1418" containerID="54bd8eba780a1c5d5060b0643278745fe759ae17570b82ce1eeb2f83696f4cfc" exitCode=0 Oct 30 19:30:09.144411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:09.144087 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" event={"ID":"7d4cdf11-f521-403d-9465-69c59fcb1418","Type":"ContainerDied","Data":"54bd8eba780a1c5d5060b0643278745fe759ae17570b82ce1eeb2f83696f4cfc"} Oct 30 19:30:10.296901 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.296873 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:10.372310 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.372267 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\") pod \"7d4cdf11-f521-403d-9465-69c59fcb1418\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " Oct 30 19:30:10.372518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.372357 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\" (UniqueName: \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\") pod \"7d4cdf11-f521-403d-9465-69c59fcb1418\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " Oct 30 19:30:10.372577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.372517 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\") pod \"7d4cdf11-f521-403d-9465-69c59fcb1418\" (UID: \"7d4cdf11-f521-403d-9465-69c59fcb1418\") " Oct 30 19:30:10.372718 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.372695 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca" pod "7d4cdf11-f521-403d-9465-69c59fcb1418" (UID: "7d4cdf11-f521-403d-9465-69c59fcb1418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:10.373072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.373051 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:10.374504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.374478 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca" pod "7d4cdf11-f521-403d-9465-69c59fcb1418" (UID: "7d4cdf11-f521-403d-9465-69c59fcb1418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:10.374607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.374550 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527" pod "7d4cdf11-f521-403d-9465-69c59fcb1418" (UID: "7d4cdf11-f521-403d-9465-69c59fcb1418"). InnerVolumeSpecName "kube-api-access-jx527". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:10.474522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.474491 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d4cdf11-f521-403d-9465-69c59fcb1418-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:10.474664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:10.474525 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx527\" (UniqueName: \"kubernetes.io/projected/7d4cdf11-f521-403d-9465-69c59fcb1418-kube-api-access-jx527\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:11.160666 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:11.160633 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" Oct 30 19:30:11.160821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:11.160664 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px" event={"ID":"7d4cdf11-f521-403d-9465-69c59fcb1418","Type":"ContainerDied","Data":"0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581"} Oct 30 19:30:11.160821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:11.160708 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c426db42779c453e0fd54ab20790ed0b2af4b23e8c8313aae03d3c10a15c581" Oct 30 19:30:13.332605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.332563 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:30:13.334083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.334052 6907 cpu_manager.go:414] "containerMap: removing container" podUID="7d4cdf11-f521-403d-9465-69c59fcb1418" containerName="revoke-keys" Oct 30 19:30:13.334222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.334208 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4cdf11-f521-403d-9465-69c59fcb1418" containerName="revoke-keys" Oct 30 19:30:13.343632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.343603 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4cdf11-f521-403d-9465-69c59fcb1418" containerName="revoke-keys" Oct 30 19:30:13.359632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.359594 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.408639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.408607 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:30:13.584065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.583968 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.607604 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607558 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr94x\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.607754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607641 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.607754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607684 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.607864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607818 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.607864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607845 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.608006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.607987 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.608099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.608083 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.608167 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.608148 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.632680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.632629 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:30:13.746293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.722743 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.746717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.746672 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.746885 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.746867 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr94x\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.746987 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.747100 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747337 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.747322 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.747464 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747752 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.747736 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.747953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.747936 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.748579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.748558 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.749060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.749038 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.749911 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.749891 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.750648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.750630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.775590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.774181 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.788282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.788204 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.833470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.808736 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.833470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.817603 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:30:13.833470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.824670 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.844086 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860242 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860424 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860579 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5tk\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860662 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860705 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860762 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860786 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.860904 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.881273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.861005 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.898231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.898138 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr94x\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:13.971469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971315 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971397 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5tk\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971474 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971513 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971567 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971595 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971670 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.971782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971748 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.972036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.971815 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.976467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.972310 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.976467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.972608 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.976467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.973631 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:13.976467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.974710 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.993294 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5tk\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.996901 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.996975 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.997888 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:13.999891 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:14.013811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:14.012386 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:14.231532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:14.214801 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:15.478698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:15.440657 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:15.878472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:15.866351 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:30:16.045152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.043382 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.045152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.043504 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.045152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.043554 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.118553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.099379 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:30:16.355475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.333591 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.355475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.333695 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.355475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.333752 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:16.449470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.448524 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:16.449470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.449329 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:30:16.449753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.449723 6907 scope.go:122] "RemoveContainer" containerID="5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1" Oct 30 19:30:16.492398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.492370 6907 scope.go:122] "RemoveContainer" containerID="bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3" Oct 30 19:30:16.507650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.507612 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:16.507944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.507908 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:16.508536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.508509 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:16.809469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.804891 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:16.809469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:16.804979 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:17.207465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.207034 6907 scope.go:122] "RemoveContainer" containerID="dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad" Oct 30 19:30:17.210333 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.209581 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:17.288061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.288013 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:17.300567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.300524 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw"] Oct 30 19:30:17.386471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.384780 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.387013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.386979 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svs5p\" (UniqueName: \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.395238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.395206 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.395513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.395490 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.395968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.390985 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw"] Oct 30 19:30:17.405006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.404983 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:17.450596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.448721 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:30:17.527050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.509473 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svs5p\" (UniqueName: \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.527050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.509535 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.527050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.509574 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.527050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.511050 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.541863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.541784 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svs5p\" (UniqueName: \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.610932 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.607802 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca035442-t78lw\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:17.637345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.612680 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" event={"ID":"b01ed4a7-3fe8-473e-97da-fa9231d4299c","Type":"ContainerStarted","Data":"b53f6548c44a000bdd7d7e4ad42b67f699fd5fc21247d5759f0eedce9f56a7ac"} Oct 30 19:30:17.645480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.645115 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" event={"ID":"06b7a44c-d384-4f50-a740-fffe46c2ed46","Type":"ContainerStarted","Data":"6d312fd1864e1718b2e6577a7e7408943935c4d119a8eb122a4761c31d87ddfe"} Oct 30 19:30:17.754030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:17.753629 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:18.362465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.362423 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:30:18.725483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.724105 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerStarted","Data":"41e2a7e67c6f1b24d1118c481cce280ac617c6dc02b33766ef5d669ee941e5dc"} Oct 30 19:30:18.738390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.738336 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" event={"ID":"b01ed4a7-3fe8-473e-97da-fa9231d4299c","Type":"ContainerStarted","Data":"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872"} Oct 30 19:30:18.741795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.741763 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" event={"ID":"06b7a44c-d384-4f50-a740-fffe46c2ed46","Type":"ContainerStarted","Data":"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432"} Oct 30 19:30:18.746427 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.746400 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerStarted","Data":"94eb02dbf40f14e03266d1771bfe3f1bb9abc74eded67d9c661b6a835cf0b19a"} Oct 30 19:30:18.750281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.750251 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerStarted","Data":"0e7e64e50f1ffdd20c8801953f1fe93dd1835813a28e6a230fab6ae7ef956dba"} Oct 30 19:30:18.780745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.777139 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podStartSLOduration=5.777117195 podStartE2EDuration="5.777117195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:18.775513148 +0000 UTC m=+624.751739094" watchObservedRunningTime="2026-10-30 19:30:18.777117195 +0000 UTC m=+624.753343141" Oct 30 19:30:18.918866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:18.918723 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podStartSLOduration=5.918701021 podStartE2EDuration="5.918701021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:18.917050408 +0000 UTC m=+624.893276353" watchObservedRunningTime="2026-10-30 19:30:18.918701021 +0000 UTC m=+624.894926956" Oct 30 19:30:19.008050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.007215 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw"] Oct 30 19:30:19.018191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.017747 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:19.018191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.017873 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:19.018191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.017918 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:19.157991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.144581 6907 scope.go:122] "RemoveContainer" containerID="c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc" Oct 30 19:30:19.239584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.239400 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:19.281838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.281809 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:19.282102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.282061 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:19.788474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.784906 6907 generic.go:356] "Generic (PLEG): container finished" podID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerID="94eb02dbf40f14e03266d1771bfe3f1bb9abc74eded67d9c661b6a835cf0b19a" exitCode=60 Oct 30 19:30:19.788474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.785105 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerDied","Data":"94eb02dbf40f14e03266d1771bfe3f1bb9abc74eded67d9c661b6a835cf0b19a"} Oct 30 19:30:19.788474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.785211 6907 scope.go:122] "RemoveContainer" containerID="dccb5a42fc77f2efc8c0076c2ea714919cd6595ae483980f9270837e7e9744ad" Oct 30 19:30:19.788474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.786003 6907 scope.go:122] "RemoveContainer" containerID="94eb02dbf40f14e03266d1771bfe3f1bb9abc74eded67d9c661b6a835cf0b19a" Oct 30 19:30:19.788474 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:19.786712 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt_odh-ai-gateway-infra(85ce9191-a5cc-4845-81b0-5a93423295e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" Oct 30 19:30:19.789756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.789660 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:19.799632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.799599 6907 generic.go:356] "Generic (PLEG): container finished" podID="5ad8080f-962e-4844-8043-e0673635bd87" containerID="0e7e64e50f1ffdd20c8801953f1fe93dd1835813a28e6a230fab6ae7ef956dba" exitCode=60 Oct 30 19:30:19.800121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.799769 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerDied","Data":"0e7e64e50f1ffdd20c8801953f1fe93dd1835813a28e6a230fab6ae7ef956dba"} Oct 30 19:30:19.800204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.800172 6907 scope.go:122] "RemoveContainer" containerID="0e7e64e50f1ffdd20c8801953f1fe93dd1835813a28e6a230fab6ae7ef956dba" Oct 30 19:30:19.800556 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:19.800530 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974_odh-ai-gateway-infra(5ad8080f-962e-4844-8043-e0673635bd87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" podUID="5ad8080f-962e-4844-8043-e0673635bd87" Oct 30 19:30:19.802303 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.802279 6907 generic.go:356] "Generic (PLEG): container finished" podID="ce971f83-6566-4031-840a-3927065301c7" containerID="41e2a7e67c6f1b24d1118c481cce280ac617c6dc02b33766ef5d669ee941e5dc" exitCode=60 Oct 30 19:30:19.803297 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.802668 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerDied","Data":"41e2a7e67c6f1b24d1118c481cce280ac617c6dc02b33766ef5d669ee941e5dc"} Oct 30 19:30:19.806022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.805983 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" event={"ID":"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981","Type":"ContainerStarted","Data":"f7904b84061a220a437c850b0021e5fe0cb7bab07341586d6e7a987eed8f5ff9"} Oct 30 19:30:19.806133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.806039 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" event={"ID":"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981","Type":"ContainerStarted","Data":"0aa73420769278645d2d757af24e9fd56829b7748935350d84a2e9adbc75f155"} Oct 30 19:30:19.811536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.811518 6907 scope.go:122] "RemoveContainer" containerID="bd0f4b46866d2a5ea59c384e3a827027f008753aee5f3e8b7352ba326ca1cbd3" Oct 30 19:30:19.850833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.846093 6907 scope.go:122] "RemoveContainer" containerID="5756463636a5811536f88b12fdaa595d3080fb8fa7cda1331eadd9885c1514e1" Oct 30 19:30:19.885586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.882349 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:19.904278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.904219 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" podStartSLOduration=3.904200134 podStartE2EDuration="3.904200134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:19.902286993 +0000 UTC m=+625.878512938" watchObservedRunningTime="2026-10-30 19:30:19.904200134 +0000 UTC m=+625.880426079" Oct 30 19:30:19.991641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:19.991605 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:20.013522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.013484 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:20.013681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.013568 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:20.197309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.197277 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:20.247618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.242411 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:20.247618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.242497 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:20.295473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.294121 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\" (UniqueName: \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\") pod \"ce971f83-6566-4031-840a-3927065301c7\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " Oct 30 19:30:20.295473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.294184 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\") pod \"ce971f83-6566-4031-840a-3927065301c7\" (UID: \"ce971f83-6566-4031-840a-3927065301c7\") " Oct 30 19:30:20.317524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.317419 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca" pod "ce971f83-6566-4031-840a-3927065301c7" (UID: "ce971f83-6566-4031-840a-3927065301c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:20.361944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.344960 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5" pod "ce971f83-6566-4031-840a-3927065301c7" (UID: "ce971f83-6566-4031-840a-3927065301c7"). InnerVolumeSpecName "kube-api-access-bzvd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:20.400639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.400207 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvd5\" (UniqueName: \"kubernetes.io/projected/ce971f83-6566-4031-840a-3927065301c7-kube-api-access-bzvd5\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.400639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.400252 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce971f83-6566-4031-840a-3927065301c7-maas-api-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:20.814256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.814219 6907 generic.go:356] "Generic (PLEG): container finished" podID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerID="1d915e220aae7f791e36f85ba66ca7b277171633e7992bdb1d1815fab6b536d7" exitCode=60 Oct 30 19:30:20.814800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.814294 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerDied","Data":"1d915e220aae7f791e36f85ba66ca7b277171633e7992bdb1d1815fab6b536d7"} Oct 30 19:30:20.814800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.814376 6907 scope.go:122] "RemoveContainer" containerID="c63e51393f951760d41eaf3eeb922f73b42be23de9eb4116244e918f2488c1fc" Oct 30 19:30:20.814800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.814735 6907 scope.go:122] "RemoveContainer" containerID="1d915e220aae7f791e36f85ba66ca7b277171633e7992bdb1d1815fab6b536d7" Oct 30 19:30:20.815033 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:20.815013 6907 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg_odh-ai-gateway-infra(3a70d09c-2dc3-4973-bd8c-89c971268f1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" Oct 30 19:30:20.816893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.816875 6907 generic.go:356] "Generic (PLEG): container finished" podID="83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" containerID="f7904b84061a220a437c850b0021e5fe0cb7bab07341586d6e7a987eed8f5ff9" exitCode=0 Oct 30 19:30:20.817004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.816991 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" event={"ID":"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981","Type":"ContainerDied","Data":"f7904b84061a220a437c850b0021e5fe0cb7bab07341586d6e7a987eed8f5ff9"} Oct 30 19:30:20.822722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.822700 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" Oct 30 19:30:20.822821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.822726 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp" event={"ID":"ce971f83-6566-4031-840a-3927065301c7","Type":"ContainerDied","Data":"92e0ae219a362c94bb0281d5ce5e7ef4705242c93fffa33fe685edd378b2e43e"} Oct 30 19:30:20.887490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.886622 6907 scope.go:122] "RemoveContainer" containerID="41e2a7e67c6f1b24d1118c481cce280ac617c6dc02b33766ef5d669ee941e5dc" Oct 30 19:30:20.996992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:20.996953 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:21.022470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.012997 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:21.022470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.013058 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:21.028580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.024557 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-f7f075-29889810-nbdfp"] Oct 30 19:30:21.293202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.291160 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:21.293202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.291250 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:21.387478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.387438 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:21.411533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.411501 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:21.522827 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.522793 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\") pod \"5ad8080f-962e-4844-8043-e0673635bd87\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " Oct 30 19:30:21.522996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.522888 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\") pod \"85ce9191-a5cc-4845-81b0-5a93423295e7\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " Oct 30 19:30:21.522996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.522948 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\" (UniqueName: \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\") pod \"85ce9191-a5cc-4845-81b0-5a93423295e7\" (UID: \"85ce9191-a5cc-4845-81b0-5a93423295e7\") " Oct 30 19:30:21.522996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.522981 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\" (UniqueName: \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\") pod \"5ad8080f-962e-4844-8043-e0673635bd87\" (UID: \"5ad8080f-962e-4844-8043-e0673635bd87\") " Oct 30 19:30:21.523945 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.523913 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca" pod "85ce9191-a5cc-4845-81b0-5a93423295e7" (UID: "85ce9191-a5cc-4845-81b0-5a93423295e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:21.524119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.523918 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca" pod "5ad8080f-962e-4844-8043-e0673635bd87" (UID: "5ad8080f-962e-4844-8043-e0673635bd87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:21.526309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.526281 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q" pod "85ce9191-a5cc-4845-81b0-5a93423295e7" (UID: "85ce9191-a5cc-4845-81b0-5a93423295e7"). InnerVolumeSpecName "kube-api-access-56p2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.534476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.526299 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7" pod "5ad8080f-962e-4844-8043-e0673635bd87" (UID: "5ad8080f-962e-4844-8043-e0673635bd87"). InnerVolumeSpecName "kube-api-access-hhfp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:21.642795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.642695 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ce9191-a5cc-4845-81b0-5a93423295e7-maas-api-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.657590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.657552 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56p2q\" (UniqueName: \"kubernetes.io/projected/85ce9191-a5cc-4845-81b0-5a93423295e7-kube-api-access-56p2q\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.657821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.657799 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhfp7\" (UniqueName: \"kubernetes.io/projected/5ad8080f-962e-4844-8043-e0673635bd87-kube-api-access-hhfp7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.657920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.657907 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ad8080f-962e-4844-8043-e0673635bd87-maas-api-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:21.834473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.834390 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" event={"ID":"5ad8080f-962e-4844-8043-e0673635bd87","Type":"ContainerDied","Data":"163fd778209a4bb68deb651aa6f4384726a37c7f30787120b432a71a47ccbeef"} Oct 30 19:30:21.834995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.834556 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974" Oct 30 19:30:21.835204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.835122 6907 scope.go:122] "RemoveContainer" containerID="0e7e64e50f1ffdd20c8801953f1fe93dd1835813a28e6a230fab6ae7ef956dba" Oct 30 19:30:21.844475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.844129 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" Oct 30 19:30:21.853316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.852919 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt" event={"ID":"85ce9191-a5cc-4845-81b0-5a93423295e7","Type":"ContainerDied","Data":"9efd557b7ecd796bb61d89beedc5d6b1ad1497fc30f69d38d57179f20c227b9c"} Oct 30 19:30:21.876038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.876004 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:21.902883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.902837 6907 scope.go:122] "RemoveContainer" containerID="94eb02dbf40f14e03266d1771bfe3f1bb9abc74eded67d9c661b6a835cf0b19a" Oct 30 19:30:21.995557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:21.995519 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:22.002623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.001029 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3bc085-29889810-w4tvt"] Oct 30 19:30:22.013314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.013280 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:22.013499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.013328 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:22.090893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.090850 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:22.110207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.110172 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-639c15-29889810-jr974"] Oct 30 19:30:22.202483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.202432 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad8080f-962e-4844-8043-e0673635bd87" path="/var/lib/kubelet/pods/5ad8080f-962e-4844-8043-e0673635bd87/volumes" Oct 30 19:30:22.203055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.203031 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" path="/var/lib/kubelet/pods/85ce9191-a5cc-4845-81b0-5a93423295e7/volumes" Oct 30 19:30:22.203621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.203600 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce971f83-6566-4031-840a-3927065301c7" path="/var/lib/kubelet/pods/ce971f83-6566-4031-840a-3927065301c7/volumes" Oct 30 19:30:22.212373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.212350 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:22.224478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.224457 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:22.287471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.259717 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:22.287769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.287719 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:22.298959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.298914 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\") pod \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " Oct 30 19:30:22.299108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.299033 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\" (UniqueName: \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\") pod \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " Oct 30 19:30:22.299108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.299071 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\" (UniqueName: \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\") pod \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " Oct 30 19:30:22.299244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.299210 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\") pod \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\" (UID: \"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981\") " Oct 30 19:30:22.299405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.299335 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\") pod \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\" (UID: \"3a70d09c-2dc3-4973-bd8c-89c971268f1c\") " Oct 30 19:30:22.300225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.300112 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca" pod "3a70d09c-2dc3-4973-bd8c-89c971268f1c" (UID: "3a70d09c-2dc3-4973-bd8c-89c971268f1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:22.300409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.300285 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a70d09c-2dc3-4973-bd8c-89c971268f1c-maas-api-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.301367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.301341 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca" pod "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" (UID: "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:22.303702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.303675 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr" pod "3a70d09c-2dc3-4973-bd8c-89c971268f1c" (UID: "3a70d09c-2dc3-4973-bd8c-89c971268f1c"). InnerVolumeSpecName "kube-api-access-l7wgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.303912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.303888 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca" pod "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" (UID: "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:22.304339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.304317 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p" pod "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" (UID: "83d00d1c-8aa9-4a5d-b4ac-e7eda727f981"). InnerVolumeSpecName "kube-api-access-svs5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:22.402295 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.402246 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.402295 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.402289 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.402577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.402308 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svs5p\" (UniqueName: \"kubernetes.io/projected/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981-kube-api-access-svs5p\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.402577 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.402323 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wgr\" (UniqueName: \"kubernetes.io/projected/3a70d09c-2dc3-4973-bd8c-89c971268f1c-kube-api-access-l7wgr\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:22.945470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.945151 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" event={"ID":"3a70d09c-2dc3-4973-bd8c-89c971268f1c","Type":"ContainerDied","Data":"32fe9981d56a49ecbba30d839f33e8aea141d98df405b754779ac9caa633267f"} Oct 30 19:30:22.945470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.945236 6907 scope.go:122] "RemoveContainer" containerID="1d915e220aae7f791e36f85ba66ca7b277171633e7992bdb1d1815fab6b536d7" Oct 30 19:30:22.945470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.945381 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg" Oct 30 19:30:22.998493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.979391 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" event={"ID":"83d00d1c-8aa9-4a5d-b4ac-e7eda727f981","Type":"ContainerDied","Data":"0aa73420769278645d2d757af24e9fd56829b7748935350d84a2e9adbc75f155"} Oct 30 19:30:22.998493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.979474 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aa73420769278645d2d757af24e9fd56829b7748935350d84a2e9adbc75f155" Oct 30 19:30:22.998493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:22.979561 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw" Oct 30 19:30:23.012849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.012823 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:23.012965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.012882 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:23.072670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.072632 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:23.083223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.083180 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-d55178-29889810-g2rvg"] Oct 30 19:30:23.244465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.234956 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:23.244465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:23.235030 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:24.013101 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.013027 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:24.013562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.013150 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:24.188128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.188081 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" path="/var/lib/kubelet/pods/3a70d09c-2dc3-4973-bd8c-89c971268f1c/volumes" Oct 30 19:30:24.240884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.240844 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:24.248597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.248550 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:24.614690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.614645 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:30:24.615618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615594 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615619 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615634 6907 cpu_manager.go:414] "containerMap: removing container" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615642 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615660 6907 cpu_manager.go:414] "containerMap: removing container" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615668 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615682 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615689 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615704 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615710 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615723 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615729 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615748 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615755 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615787 6907 cpu_manager.go:414] "containerMap: removing container" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615795 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615812 6907 cpu_manager.go:414] "containerMap: removing container" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615820 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615839 6907 cpu_manager.go:414] "containerMap: removing container" podUID="83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" containerName="revoke-keys" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615846 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" containerName="revoke-keys" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615868 6907 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.615884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.615875 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616019 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616033 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616044 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616056 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" containerName="revoke-keys" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616067 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616079 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616089 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616100 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616108 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.616783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.616118 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce971f83-6566-4031-840a-3927065301c7" containerName="cleanup" Oct 30 19:30:24.622646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.622616 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.641132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.641070 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"distinct-w2-d55178-kserve-self-signed-certs\"" Oct 30 19:30:24.652388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.651486 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787249 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787335 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787481 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787630 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787794 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.791473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.787864 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6m9\" (UniqueName: \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.893424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.893341 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.893818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.893793 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.893972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.893958 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.894099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.894087 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.894231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.894213 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6m9\" (UniqueName: \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.895099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.894722 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.895099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.894856 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.895404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.895132 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.895583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.895434 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.928281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.922367 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.939509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.939430 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.939839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.939797 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6m9\" (UniqueName: \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\") pod \"distinct-w2-d55178-kserve-fc4c9bfc-q4vpv\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:24.966323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:24.966221 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:25.015731 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:25.015675 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:30:25.016257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:25.015757 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 19:30:25.232893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:25.232854 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:30:25.233113 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:25.232917 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 19:30:25.285674 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:25.285636 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:30:26.020233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.020203 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:26.020685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.020661 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:26.021548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.021529 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:30:26.067578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.067543 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" event={"ID":"24f80684-9767-4cd0-a7c9-c8058ab5b7fb","Type":"ContainerStarted","Data":"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a"} Oct 30 19:30:26.067748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.067599 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" event={"ID":"24f80684-9767-4cd0-a7c9-c8058ab5b7fb","Type":"ContainerStarted","Data":"6213b68b8593cafeb61f4384a32985eb9c50e6394c2f242dc78839e3732453fe"} Oct 30 19:30:26.089131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.089062 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podStartSLOduration=2.089041655 podStartE2EDuration="2.089041655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:26.088169142 +0000 UTC m=+632.064395106" watchObservedRunningTime="2026-10-30 19:30:26.089041655 +0000 UTC m=+632.065267601" Oct 30 19:30:26.236657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.236626 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:26.237121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.237109 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:26.237905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.237885 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:30:26.966934 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:26.966894 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:27.950221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.950188 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:30:27.951005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.950984 6907 cpu_manager.go:414] "containerMap: removing container" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:27.951005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951005 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:27.951120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951043 6907 cpu_manager.go:414] "containerMap: removing container" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:27.951120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951049 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:27.951180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951151 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad8080f-962e-4844-8043-e0673635bd87" containerName="cleanup" Oct 30 19:30:27.951180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951165 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a70d09c-2dc3-4973-bd8c-89c971268f1c" containerName="cleanup" Oct 30 19:30:27.951180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.951172 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ce9191-a5cc-4845-81b0-5a93423295e7" containerName="cleanup" Oct 30 19:30:27.954990 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.954972 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:27.977923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:27.977888 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:30:28.018382 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.018341 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:30:28.023944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.023916 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.056964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.056855 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:30:28.141602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141567 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.141808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141624 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.141808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141703 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.141808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141756 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z6g9\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.141808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141794 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.141966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141871 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.141966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141903 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcx5\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.141966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.141931 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142052 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142083 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142150 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142223 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.142303 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142263 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.142353 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142301 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.142422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142405 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142457 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.142547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142499 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.142547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.142524 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.243972 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244010 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244036 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z6g9\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244072 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244118 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244281 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcx5\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244361 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244471 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244482 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244514 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244563 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244627 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244710 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244746 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244750 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244841 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244876 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.244956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244922 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.244956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.244948 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.245102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245031 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.245285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245255 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.245395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245288 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.245395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245262 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.245565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245544 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.245772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245711 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.245772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.245722 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.246035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.246012 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.246602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.246424 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.246860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.246828 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.247257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.247233 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.247562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.247543 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.247650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.247621 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.253418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.253396 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.253623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.253602 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcx5\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.254317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.254296 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.254601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.254582 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z6g9\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.275022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.274991 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:28.346089 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.345752 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:28.619359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.619325 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:30:28.620213 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:28.620184 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56c4323_7158_4170_9d3a_81256d394f1a.slice/crio-1d29c3496a52b898f532c29d21a69e8ad5ba8edbbed356e531bfcf1b3a9c9a15 WatchSource:0}: Error finding container 1d29c3496a52b898f532c29d21a69e8ad5ba8edbbed356e531bfcf1b3a9c9a15: Status 404 returned error can't find the container with id 1d29c3496a52b898f532c29d21a69e8ad5ba8edbbed356e531bfcf1b3a9c9a15 Oct 30 19:30:28.623055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.623021 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.623173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.623106 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.623173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.623155 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.703987 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:28.703957 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72faf153_3f73_433a_9122_a3167466366c.slice/crio-eac69892b830fdf036e3e2f57ea084485a7076b65bbebd7ad4c0b2d573725337 WatchSource:0}: Error finding container eac69892b830fdf036e3e2f57ea084485a7076b65bbebd7ad4c0b2d573725337: Status 404 returned error can't find the container with id eac69892b830fdf036e3e2f57ea084485a7076b65bbebd7ad4c0b2d573725337 Oct 30 19:30:28.704687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.704663 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:30:28.706581 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.706550 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.706656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.706615 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:28.706656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:28.706646 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:29.102126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.102087 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" event={"ID":"72faf153-3f73-433a-9122-a3167466366c","Type":"ContainerStarted","Data":"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914"} Oct 30 19:30:29.102646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.102153 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" event={"ID":"72faf153-3f73-433a-9122-a3167466366c","Type":"ContainerStarted","Data":"eac69892b830fdf036e3e2f57ea084485a7076b65bbebd7ad4c0b2d573725337"} Oct 30 19:30:29.104236 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.104200 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" event={"ID":"a56c4323-7158-4170-9d3a-81256d394f1a","Type":"ContainerStarted","Data":"a227ce00920c2cb7e5ebba32fb111491c31fb1a854ae66882ba7eba6cd838a38"} Oct 30 19:30:29.104355 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.104266 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" event={"ID":"a56c4323-7158-4170-9d3a-81256d394f1a","Type":"ContainerStarted","Data":"1d29c3496a52b898f532c29d21a69e8ad5ba8edbbed356e531bfcf1b3a9c9a15"} Oct 30 19:30:29.125317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.125257 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podStartSLOduration=2.1252420770000002 podStartE2EDuration="2.125242077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:29.123374312 +0000 UTC m=+635.099600258" watchObservedRunningTime="2026-10-30 19:30:29.125242077 +0000 UTC m=+635.101468021" Oct 30 19:30:29.149597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.149536 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podStartSLOduration=2.149514432 podStartE2EDuration="2.149514432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:29.147631508 +0000 UTC m=+635.123857453" watchObservedRunningTime="2026-10-30 19:30:29.149514432 +0000 UTC m=+635.125740378" Oct 30 19:30:29.215777 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.215726 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl"] Oct 30 19:30:29.222031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.221997 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.226174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.226144 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:29.226328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.226294 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:30:29.227836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.227801 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl"] Oct 30 19:30:29.276157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.276118 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:29.276396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.276291 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:29.276554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.276531 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:29.285431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.285387 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnp6\" (UniqueName: \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.285617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.285569 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.285708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.285691 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.367127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.367027 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:29.374846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.372196 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:29.374846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.372258 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:29.404160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.395419 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnp6\" (UniqueName: \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.413469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.404596 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.413906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.413854 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.416791 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.416284 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.420619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.420587 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.452465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.451144 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnp6\" (UniqueName: \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\") pod \"maas-api-revoke-sub-b2464616-lb7tl\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.542727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.542688 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:29.901031 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:29.900998 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f045a0_31c7_4737_879b_44b21b296caa.slice/crio-e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1 WatchSource:0}: Error finding container e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1: Status 404 returned error can't find the container with id e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1 Oct 30 19:30:29.901411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:29.901386 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl"] Oct 30 19:30:30.114761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:30.114711 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" event={"ID":"a1f045a0-31c7-4737-879b-44b21b296caa","Type":"ContainerStarted","Data":"e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1"} Oct 30 19:30:30.280777 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:30.280736 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:30.281029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:30.280999 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:30.374761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:30.374519 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:30.374761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:30.374579 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:31.124108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.124065 6907 generic.go:356] "Generic (PLEG): container finished" podID="a1f045a0-31c7-4737-879b-44b21b296caa" containerID="f88179c9a08e8c8c7b573aee756b2a3eaf04d15d3a2c4574672e0c0ef2198390" exitCode=0 Oct 30 19:30:31.124705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.124110 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" event={"ID":"a1f045a0-31c7-4737-879b-44b21b296caa","Type":"ContainerDied","Data":"f88179c9a08e8c8c7b573aee756b2a3eaf04d15d3a2c4574672e0c0ef2198390"} Oct 30 19:30:31.276004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.275874 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:31.276004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.275938 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:31.378067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.378029 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:31.378361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:31.378328 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:32.276002 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.275970 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:32.276514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.276485 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:32.385863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.385368 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:32.385863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.385431 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:32.423340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.423311 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:32.675907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.675858 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\") pod \"a1f045a0-31c7-4737-879b-44b21b296caa\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " Oct 30 19:30:32.676139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.676038 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\" (UniqueName: \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\") pod \"a1f045a0-31c7-4737-879b-44b21b296caa\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " Oct 30 19:30:32.676139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.676078 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\") pod \"a1f045a0-31c7-4737-879b-44b21b296caa\" (UID: \"a1f045a0-31c7-4737-879b-44b21b296caa\") " Oct 30 19:30:32.680727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.680679 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca" pod "a1f045a0-31c7-4737-879b-44b21b296caa" (UID: "a1f045a0-31c7-4737-879b-44b21b296caa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:32.685513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.685481 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca" pod "a1f045a0-31c7-4737-879b-44b21b296caa" (UID: "a1f045a0-31c7-4737-879b-44b21b296caa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:32.685835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.685814 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6" pod "a1f045a0-31c7-4737-879b-44b21b296caa" (UID: "a1f045a0-31c7-4737-879b-44b21b296caa"). InnerVolumeSpecName "kube-api-access-fqnp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:32.777866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.777818 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:32.777866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.777857 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnp6\" (UniqueName: \"kubernetes.io/projected/a1f045a0-31c7-4737-879b-44b21b296caa-kube-api-access-fqnp6\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:32.777866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:32.777874 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1f045a0-31c7-4737-879b-44b21b296caa-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:33.079064 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.079005 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:30:33.194236 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.194198 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" event={"ID":"a1f045a0-31c7-4737-879b-44b21b296caa","Type":"ContainerDied","Data":"e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1"} Oct 30 19:30:33.194433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.194260 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e456c74d34524bd4d954ff984edf13aa6b2b5340d46637a198e332e45e15e2d1" Oct 30 19:30:33.194433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.194276 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl" Oct 30 19:30:33.276470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.276158 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:33.276470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.276252 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:33.368530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.367988 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:33.368530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:33.368048 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:34.279939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:34.279896 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.280554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:34.280516 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:34.379467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:34.370865 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:34.379467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:34.371017 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:34.997741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:34.997690 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:35.279063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:35.278962 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.279349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:35.279315 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:35.366716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:35.366659 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:35.367327 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:35.367294 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:35.531026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:35.530925 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:36.280470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.278364 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.280470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.278429 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:30:36.372492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.367939 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:36.372492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.368020 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:36.509467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.508071 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:36.536527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:36.535559 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.385467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.384906 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:30:37.385467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.385000 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 19:30:37.430481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.417043 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:30:37.430481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.426159 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:37.430481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.426220 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.538586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.538544 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:37.553770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.538825 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:37.888290 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:37.888216 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:30:37.897193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.895638 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:37.898048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.897861 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:37.899121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:37.899060 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:30:38.604406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.604366 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:30:38.609158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.607294 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a1f045a0-31c7-4737-879b-44b21b296caa" containerName="revoke-keys" Oct 30 19:30:38.609158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.607318 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f045a0-31c7-4737-879b-44b21b296caa" containerName="revoke-keys" Oct 30 19:30:38.609158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.607506 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f045a0-31c7-4737-879b-44b21b296caa" containerName="revoke-keys" Oct 30 19:30:38.615060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.615030 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:30:38.615183 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.615155 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.619386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.619362 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3bc085\"/\"default-dockercfg-bhsf7\"" Oct 30 19:30:38.619545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.619363 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3bc085\"/\"distinct-w4-3bc085-kserve-self-signed-certs\"" Oct 30 19:30:38.619545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.619529 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3bc085\"/\"openshift-service-ca.crt\"" Oct 30 19:30:38.619664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.619608 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3bc085\"/\"kube-root-ca.crt\"" Oct 30 19:30:38.681695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.681650 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.682593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.682375 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.682593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.682531 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.682856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.682611 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729jt\" (UniqueName: \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.682856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.682677 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.682856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.682717 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.783683 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.783741 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.783859 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.783923 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729jt\" (UniqueName: \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.783976 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.784011 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.785004 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.785539 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.790468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.786171 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.830859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.828438 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.830859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.829122 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.842481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.836832 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729jt\" (UniqueName: \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\") pod \"distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:38.939476 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:38.939341 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:39.235529 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:39.235497 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e40913e_6543_401e_b90d_402694f7dc1e.slice/crio-967e675eaf702a72953d9de97d92f52f515384e6f3016e32e3393afe06a4435e WatchSource:0}: Error finding container 967e675eaf702a72953d9de97d92f52f515384e6f3016e32e3393afe06a4435e: Status 404 returned error can't find the container with id 967e675eaf702a72953d9de97d92f52f515384e6f3016e32e3393afe06a4435e Oct 30 19:30:39.237825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.237796 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:30:39.408721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.408601 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:30:39.408721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.408670 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:30:39.436342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.436300 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" event={"ID":"8e40913e-6543-401e-b90d-402694f7dc1e","Type":"ContainerStarted","Data":"073b8158edec3c5e6b890361a90bc4c56c7d8a1249ad1bb248465d44c1f2a7db"} Oct 30 19:30:39.436590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.436355 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" event={"ID":"8e40913e-6543-401e-b90d-402694f7dc1e","Type":"ContainerStarted","Data":"967e675eaf702a72953d9de97d92f52f515384e6f3016e32e3393afe06a4435e"} Oct 30 19:30:39.476562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.476500 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podStartSLOduration=1.476484586 podStartE2EDuration="1.476484586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:39.474886665 +0000 UTC m=+645.451112624" watchObservedRunningTime="2026-10-30 19:30:39.476484586 +0000 UTC m=+645.452710530" Oct 30 19:30:39.939918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:39.939875 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:40.063937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:40.063899 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:40.064515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:40.064487 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:40.065514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:40.065489 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:30:42.986770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:42.986735 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:30:42.991328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:42.991305 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.004870 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.004838 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:30:43.053266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053228 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053455 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053389 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053455 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053523 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053581 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7scs\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053686 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053751 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053844 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.053894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.053881 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155644 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155700 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155724 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155757 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7scs\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155801 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.155894 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155882 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155929 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.155979 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156023 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156104 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156144 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156431 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156474 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.156741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.156716 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.158526 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.158502 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.158873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.158852 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.165329 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.165302 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7scs\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.165484 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.165347 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.209759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.209721 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:30:43.218904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.218861 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.225608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.225577 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:30:43.257904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.257823 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.257904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.257880 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.257968 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258027 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258055 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258234 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258341 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258375 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tk2\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.258493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.258413 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.309534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.309503 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:43.359507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359466 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.359507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359515 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.359767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359572 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.359767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359614 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.359767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359640 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tk2\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.359767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359667 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359782 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359839 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359868 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.359954 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.360256 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.360398 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.360432 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.360719 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.360700 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.362240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.362218 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.362436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.362404 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.370519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.370495 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.370710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.370691 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tk2\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.552858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.552763 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:43.604351 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.604300 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:30:43.604728 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:43.604699 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a7879a_cccd_4d31_b5f2_d56cafd48616.slice/crio-798abc37fa76b93f435e1a341b2298431a5a9abc6a3943590d5cde79b5f4a3ba WatchSource:0}: Error finding container 798abc37fa76b93f435e1a341b2298431a5a9abc6a3943590d5cde79b5f4a3ba: Status 404 returned error can't find the container with id 798abc37fa76b93f435e1a341b2298431a5a9abc6a3943590d5cde79b5f4a3ba Oct 30 19:30:43.638320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.636578 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:43.638320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.636680 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:43.638320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.636725 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:43.873743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.873716 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:30:43.877087 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:43.877046 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b83a86_3762_4388_ae94_ae19a3dbabe2.slice/crio-ad8c2608adc03adc23e6751681614759a4611abe99aeb4dd9cc08aaf5fea0946 WatchSource:0}: Error finding container ad8c2608adc03adc23e6751681614759a4611abe99aeb4dd9cc08aaf5fea0946: Status 404 returned error can't find the container with id ad8c2608adc03adc23e6751681614759a4611abe99aeb4dd9cc08aaf5fea0946 Oct 30 19:30:43.880359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.880317 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:43.880468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.880409 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:43.880513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:43.880474 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:44.489034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.488986 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" event={"ID":"09b83a86-3762-4388-ae94-ae19a3dbabe2","Type":"ContainerStarted","Data":"2fad8aea7b8f5081a83bb8792d5facb3f019fd1f9c0e6ea00c4139387797c361"} Oct 30 19:30:44.489483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.489061 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" event={"ID":"09b83a86-3762-4388-ae94-ae19a3dbabe2","Type":"ContainerStarted","Data":"ad8c2608adc03adc23e6751681614759a4611abe99aeb4dd9cc08aaf5fea0946"} Oct 30 19:30:44.490976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.490953 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" event={"ID":"60a7879a-cccd-4d31-b5f2-d56cafd48616","Type":"ContainerStarted","Data":"2d0a62ac1f0ebd9b390af02dc6cf2f4f0f1da83653665edd22fbee6ba4391509"} Oct 30 19:30:44.491026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.491001 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" event={"ID":"60a7879a-cccd-4d31-b5f2-d56cafd48616","Type":"ContainerStarted","Data":"798abc37fa76b93f435e1a341b2298431a5a9abc6a3943590d5cde79b5f4a3ba"} Oct 30 19:30:44.550046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.549984 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podStartSLOduration=2.5499649890000002 podStartE2EDuration="2.549964989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:44.549515487 +0000 UTC m=+650.525741431" watchObservedRunningTime="2026-10-30 19:30:44.549964989 +0000 UTC m=+650.526190938" Oct 30 19:30:44.552810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.552748 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podStartSLOduration=1.552733082 podStartE2EDuration="1.552733082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:44.516621089 +0000 UTC m=+650.492847033" watchObservedRunningTime="2026-10-30 19:30:44.552733082 +0000 UTC m=+650.528959029" Oct 30 19:30:44.561888 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.561808 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:44.564564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.564192 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:44.564564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:44.564257 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:45.309664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:45.309626 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:45.310966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:45.310942 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:45.311124 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:45.311104 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:45.569616 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:45.567541 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:45.569616 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:45.567606 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:46.310618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:46.310581 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:46.310800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:46.310636 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:46.562692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:46.562592 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:46.564697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:46.564659 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:47.310034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:47.309997 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:47.310431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:47.310057 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:47.557960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:47.557919 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:47.558256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:47.558223 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:48.309915 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.309881 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:48.310165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.310141 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:48.555469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.554828 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:48.555469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.554905 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:48.943969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.943919 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:48.990218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:48.990187 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:49.309931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:49.309896 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:49.310131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:49.309954 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:49.553832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:49.553792 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:49.554403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:49.554370 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:49.600985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:49.600903 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:30:50.310398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:50.310356 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 19:30:50.310610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:50.310439 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 19:30:50.555059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:50.555023 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:30:50.555737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:50.555701 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 19:30:51.330289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.330254 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:51.330641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.330618 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:51.331392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.331372 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:30:51.592530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.591845 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:51.593426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.593403 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:51.601560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:51.601529 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:30:53.024890 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.024854 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc"] Oct 30 19:30:53.031012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.030984 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.036072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.035524 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:30:53.037899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.037864 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc"] Oct 30 19:30:53.038080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.038063 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:30:53.069842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.069803 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.070005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.069957 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wll7g\" (UniqueName: \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.070081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.070002 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.170955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.170900 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wll7g\" (UniqueName: \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.171162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.170977 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.171162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.171120 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.171688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.171657 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.173493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.173470 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.180209 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.180182 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wll7g\" (UniqueName: \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\") pod \"maas-api-revoke-sub-e1dc9336-nh2hc\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.350863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.350771 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:53.534660 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.533945 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:30:53.553512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.553475 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:30:53.553708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.553585 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.559217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.559045 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-639c15\"/\"kube-root-ca.crt\"" Oct 30 19:30:53.559217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.559196 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-639c15\"/\"default-dockercfg-jxglj\"" Oct 30 19:30:53.559562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.559243 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-639c15\"/\"openshift-service-ca.crt\"" Oct 30 19:30:53.559562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.559348 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-639c15\"/\"distinct-w6-639c15-kserve-self-signed-certs\"" Oct 30 19:30:53.617591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.617492 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.617591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.617533 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.617814 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.617661 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.617883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.617862 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.618032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.618013 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2t4\" (UniqueName: \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.618106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.618050 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.719362 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.719317 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.719626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.719603 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.719839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.719823 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.720127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.720099 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.720467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.720426 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2t4\" (UniqueName: \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.721400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.721370 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.726025 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.725908 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.726208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.724653 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.726349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.724191 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.756489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.738817 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.756489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.741773 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2t4\" (UniqueName: \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.756489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.749589 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc"] Oct 30 19:30:53.766554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.765343 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\") pod \"distinct-w6-639c15-kserve-6c8c4647f8-mg7dq\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:53.873132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:53.873051 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:54.164216 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.164192 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:30:54.166010 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:54.165986 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56b73626_6cd2_4d5c_bdea_abc9efe3f2b7.slice/crio-6d0de7cfcc7371119cabc580a01cf6d4516c3559da51cb8d08e636ceaf72daf3 WatchSource:0}: Error finding container 6d0de7cfcc7371119cabc580a01cf6d4516c3559da51cb8d08e636ceaf72daf3: Status 404 returned error can't find the container with id 6d0de7cfcc7371119cabc580a01cf6d4516c3559da51cb8d08e636ceaf72daf3 Oct 30 19:30:54.773653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.773620 6907 generic.go:356] "Generic (PLEG): container finished" podID="a2902e97-612a-491d-aab8-ea1c6cf83881" containerID="68ae856e8792f52ec7afe531abda374b748b809549e86ccf54e03d41cfcae0e0" exitCode=0 Oct 30 19:30:54.773834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.773658 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" event={"ID":"a2902e97-612a-491d-aab8-ea1c6cf83881","Type":"ContainerDied","Data":"68ae856e8792f52ec7afe531abda374b748b809549e86ccf54e03d41cfcae0e0"} Oct 30 19:30:54.773834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.773709 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" event={"ID":"a2902e97-612a-491d-aab8-ea1c6cf83881","Type":"ContainerStarted","Data":"d5d345358f59fd59e6cfaa2d45345ad878153c97d64aefb27a1b27c0139c469c"} Oct 30 19:30:54.775359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.775337 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" event={"ID":"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7","Type":"ContainerStarted","Data":"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7"} Oct 30 19:30:54.775359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.775367 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" event={"ID":"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7","Type":"ContainerStarted","Data":"6d0de7cfcc7371119cabc580a01cf6d4516c3559da51cb8d08e636ceaf72daf3"} Oct 30 19:30:54.820089 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.820033 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podStartSLOduration=1.820017722 podStartE2EDuration="1.820017722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:54.815705647 +0000 UTC m=+660.791931592" watchObservedRunningTime="2026-10-30 19:30:54.820017722 +0000 UTC m=+660.796243685" Oct 30 19:30:54.873927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:54.873891 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:30:56.331469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.317894 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:56.399074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.399029 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\") pod \"a2902e97-612a-491d-aab8-ea1c6cf83881\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " Oct 30 19:30:56.399962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.399930 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\" (UniqueName: \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\") pod \"a2902e97-612a-491d-aab8-ea1c6cf83881\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " Oct 30 19:30:56.400202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.400184 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\") pod \"a2902e97-612a-491d-aab8-ea1c6cf83881\" (UID: \"a2902e97-612a-491d-aab8-ea1c6cf83881\") " Oct 30 19:30:56.401483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.401434 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca" pod "a2902e97-612a-491d-aab8-ea1c6cf83881" (UID: "a2902e97-612a-491d-aab8-ea1c6cf83881"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:56.503561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.503263 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.534912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.505574 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca" pod "a2902e97-612a-491d-aab8-ea1c6cf83881" (UID: "a2902e97-612a-491d-aab8-ea1c6cf83881"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.534912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.507621 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g" pod "a2902e97-612a-491d-aab8-ea1c6cf83881" (UID: "a2902e97-612a-491d-aab8-ea1c6cf83881"). InnerVolumeSpecName "kube-api-access-wll7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:56.607056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.604569 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2902e97-612a-491d-aab8-ea1c6cf83881-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.607056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.604604 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wll7g\" (UniqueName: \"kubernetes.io/projected/a2902e97-612a-491d-aab8-ea1c6cf83881-kube-api-access-wll7g\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.872635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.872546 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" Oct 30 19:30:56.873019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.872989 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc" event={"ID":"a2902e97-612a-491d-aab8-ea1c6cf83881","Type":"ContainerDied","Data":"d5d345358f59fd59e6cfaa2d45345ad878153c97d64aefb27a1b27c0139c469c"} Oct 30 19:30:56.873223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:56.873204 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d345358f59fd59e6cfaa2d45345ad878153c97d64aefb27a1b27c0139c469c" Oct 30 19:30:57.207525 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:30:57.204766 6907 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2902e97_612a_491d_aab8_ea1c6cf83881.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:30:58.131708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.131658 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:30:58.132761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.132732 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a2902e97-612a-491d-aab8-ea1c6cf83881" containerName="revoke-keys" Oct 30 19:30:58.132761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.132763 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2902e97-612a-491d-aab8-ea1c6cf83881" containerName="revoke-keys" Oct 30 19:30:58.132950 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.132941 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2902e97-612a-491d-aab8-ea1c6cf83881" containerName="revoke-keys" Oct 30 19:30:58.137554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.137527 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:30:58.137808 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.137788 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.199985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.199943 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkbf\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200156 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200006 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200156 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200129 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200258 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200301 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200330 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200465 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200502 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.200627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.200547 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.308467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308398 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.308683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308553 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.308683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308617 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.308846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308776 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkbf\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.308909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308844 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.308991 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.309173 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.309242 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.309276 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.309519 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.309807 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.310023 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.312465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.310234 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.336537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.323242 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.336537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.333079 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.360360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.350787 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.360774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.360741 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkbf\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.383510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.382118 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.464468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.449212 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:30:58.484468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.479138 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:30:58.484468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.479276 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.496557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.493009 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:30:58.620223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620188 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620222 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620385 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620417 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzlgw\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620462 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620498 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620533 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620675 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.620737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.620713 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722204 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722246 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzlgw\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722378 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722460 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722514 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722736 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722769 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722793 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722822 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722852 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722882 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.722989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.722900 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.723167 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.723144 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.723835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.723799 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.724753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.724724 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.725378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.725353 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.747506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.747475 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.747646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.747628 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzlgw\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.858029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.857995 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:30:58.858622 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:58.858593 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b04c47_f047_4cc6_9ad1_2ca346155ec0.slice/crio-769ee81a5bdc1bfd4a3d29809f9727086557edc509b181ed181eb29175dd70e6 WatchSource:0}: Error finding container 769ee81a5bdc1bfd4a3d29809f9727086557edc509b181ed181eb29175dd70e6: Status 404 returned error can't find the container with id 769ee81a5bdc1bfd4a3d29809f9727086557edc509b181ed181eb29175dd70e6 Oct 30 19:30:58.861231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.861199 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:58.861304 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.861267 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:58.861304 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.861298 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:58.863327 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.863305 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:30:58.898879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:58.898846 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" event={"ID":"56b04c47-f047-4cc6-9ad1-2ca346155ec0","Type":"ContainerStarted","Data":"769ee81a5bdc1bfd4a3d29809f9727086557edc509b181ed181eb29175dd70e6"} Oct 30 19:30:59.162478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.162429 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:30:59.163084 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:30:59.163055 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e39e7c_afbd_4f29_8387_83fb07385c2b.slice/crio-6526d985e0ad7131785c12c8c6a5eff92cfc13f2c08b3ab49478f36ceb807b4d WatchSource:0}: Error finding container 6526d985e0ad7131785c12c8c6a5eff92cfc13f2c08b3ab49478f36ceb807b4d: Status 404 returned error can't find the container with id 6526d985e0ad7131785c12c8c6a5eff92cfc13f2c08b3ab49478f36ceb807b4d Oct 30 19:30:59.165776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.165742 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:59.165899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.165811 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:59.165899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.165842 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:30:59.907842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.907800 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" event={"ID":"58e39e7c-afbd-4f29-8387-83fb07385c2b","Type":"ContainerStarted","Data":"d4f344e11cb38fa5d78f717c752814572286fd1cc4abf901f3f75b3e425f2096"} Oct 30 19:30:59.908079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.908049 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" event={"ID":"58e39e7c-afbd-4f29-8387-83fb07385c2b","Type":"ContainerStarted","Data":"6526d985e0ad7131785c12c8c6a5eff92cfc13f2c08b3ab49478f36ceb807b4d"} Oct 30 19:30:59.909667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.909641 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" event={"ID":"56b04c47-f047-4cc6-9ad1-2ca346155ec0","Type":"ContainerStarted","Data":"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7"} Oct 30 19:30:59.984806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:30:59.984732 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podStartSLOduration=1.9847126990000001 podStartE2EDuration="1.984712699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:59.982985862 +0000 UTC m=+665.959211809" watchObservedRunningTime="2026-10-30 19:30:59.984712699 +0000 UTC m=+665.960938645" Oct 30 19:31:00.011012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.010963 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podStartSLOduration=2.01094744 podStartE2EDuration="2.01094744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:00.009685273 +0000 UTC m=+665.985911218" watchObservedRunningTime="2026-10-30 19:31:00.01094744 +0000 UTC m=+665.987173384" Oct 30 19:31:00.511474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.500691 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:31:00.511474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.502555 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:00.511474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.502610 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:00.792514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.792390 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:31:00.799296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.799264 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.799771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.799746 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:31:00.805062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.805034 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d55178\"/\"distinct-2-w2-d55178-kserve-self-signed-certs\"" Oct 30 19:31:00.882515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.869521 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:31:00.882998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.882955 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:00.883113 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.883050 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.905844 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.905972 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.906078 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.906191 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.906249 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:00.922468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:00.906303 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lhf\" (UniqueName: \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008425 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008668 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008734 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008792 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lhf\" (UniqueName: \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008852 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.008960 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.016145 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.017306 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.020477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.017638 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.068340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.068255 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lhf\" (UniqueName: \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.111657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.111004 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.112118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.112063 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\") pod \"distinct-2-w2-d55178-kserve-64f478dc54-dgzjh\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.118553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.118213 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:01.494521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.494477 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:01.494734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.494548 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:01.520515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.520488 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:31:01.521298 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:01.521269 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11df2f79_da70_47f0_b25e_0de02c207bc8.slice/crio-4319909cc33af71826b4dbce88844801f43e05b8865e6c322f018665f4fbf049 WatchSource:0}: Error finding container 4319909cc33af71826b4dbce88844801f43e05b8865e6c322f018665f4fbf049: Status 404 returned error can't find the container with id 4319909cc33af71826b4dbce88844801f43e05b8865e6c322f018665f4fbf049 Oct 30 19:31:01.870520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.868934 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:01.870520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:01.869005 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:02.086270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.086226 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" event={"ID":"11df2f79-da70-47f0-b25e-0de02c207bc8","Type":"ContainerStarted","Data":"cc3549ccc5180a219611eac33806a38d3cf48f93108ee50c10dfea7d2a74acbd"} Oct 30 19:31:02.086465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.086284 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" event={"ID":"11df2f79-da70-47f0-b25e-0de02c207bc8","Type":"ContainerStarted","Data":"4319909cc33af71826b4dbce88844801f43e05b8865e6c322f018665f4fbf049"} Oct 30 19:31:02.108201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.108151 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podStartSLOduration=2.108136875 podStartE2EDuration="2.108136875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:02.104112546 +0000 UTC m=+668.080338501" watchObservedRunningTime="2026-10-30 19:31:02.108136875 +0000 UTC m=+668.084362820" Oct 30 19:31:02.119705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.119679 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:02.494613 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.494580 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:02.494806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.494632 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:02.865523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.864872 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:02.865523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:02.864938 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:03.494567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.494516 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:03.494755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.494585 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:03.864026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.863935 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:03.864325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.864294 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:03.874826 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.874793 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:03.919603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:03.919567 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:04.113893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:04.113866 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:04.495406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:04.495370 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:04.495620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:04.495433 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:04.863996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:04.863897 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:04.864274 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:04.864241 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:05.495082 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:05.495045 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 19:31:05.495527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:05.495109 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 19:31:05.868953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:05.868853 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:31:05.869235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:05.869200 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 19:31:06.663094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.663062 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:31:06.663592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.663567 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:31:06.664562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.664539 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:31:06.968817 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.968726 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:31:06.969547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.969522 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:31:06.972591 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:06.972563 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:31:07.207328 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.207291 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9"] Oct 30 19:31:07.213168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.213143 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.217548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.217523 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:07.217702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.217677 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:31:07.220229 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.220208 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9"] Oct 30 19:31:07.286868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.286828 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.287129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.287098 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.287256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.287227 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpwp\" (UniqueName: \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.388331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.388296 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.388576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.388391 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.388576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.388420 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpwp\" (UniqueName: \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.388940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.388919 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.390780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.390759 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.397637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.397615 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpwp\" (UniqueName: \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\") pod \"maas-api-revoke-sub-164203ac-6t9z9\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.532767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.532685 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:07.772171 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:31:07.772134 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:31:07.814530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:07.814500 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9"] Oct 30 19:31:07.815365 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:07.815339 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca787e75_dcf6_4450_9725_406d89f8dd6f.slice/crio-e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81 WatchSource:0}: Error finding container e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81: Status 404 returned error can't find the container with id e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81 Oct 30 19:31:08.169711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:08.169572 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" event={"ID":"ca787e75-dcf6-4450-9725-406d89f8dd6f","Type":"ContainerStarted","Data":"e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81"} Oct 30 19:31:09.180215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:09.180111 6907 generic.go:356] "Generic (PLEG): container finished" podID="ca787e75-dcf6-4450-9725-406d89f8dd6f" containerID="bb94f1b2f0393dff234ea2440c25951fe8efc26d77130fa608c109dba2548384" exitCode=0 Oct 30 19:31:09.180755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:09.180211 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" event={"ID":"ca787e75-dcf6-4450-9725-406d89f8dd6f","Type":"ContainerDied","Data":"bb94f1b2f0393dff234ea2440c25951fe8efc26d77130fa608c109dba2548384"} Oct 30 19:31:10.416027 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.416003 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:10.549171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.549137 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\") pod \"ca787e75-dcf6-4450-9725-406d89f8dd6f\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " Oct 30 19:31:10.549366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.549286 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\" (UniqueName: \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\") pod \"ca787e75-dcf6-4450-9725-406d89f8dd6f\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " Oct 30 19:31:10.549366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.549343 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\") pod \"ca787e75-dcf6-4450-9725-406d89f8dd6f\" (UID: \"ca787e75-dcf6-4450-9725-406d89f8dd6f\") " Oct 30 19:31:10.549801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.549772 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca" pod "ca787e75-dcf6-4450-9725-406d89f8dd6f" (UID: "ca787e75-dcf6-4450-9725-406d89f8dd6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:10.551320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.551297 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca" pod "ca787e75-dcf6-4450-9725-406d89f8dd6f" (UID: "ca787e75-dcf6-4450-9725-406d89f8dd6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:10.551396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.551363 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp" pod "ca787e75-dcf6-4450-9725-406d89f8dd6f" (UID: "ca787e75-dcf6-4450-9725-406d89f8dd6f"). InnerVolumeSpecName "kube-api-access-gjpwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:10.650822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.650791 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:10.650822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.650819 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjpwp\" (UniqueName: \"kubernetes.io/projected/ca787e75-dcf6-4450-9725-406d89f8dd6f-kube-api-access-gjpwp\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:10.650822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:10.650830 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca787e75-dcf6-4450-9725-406d89f8dd6f-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:11.124512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.120976 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:11.141705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.141675 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:11.197038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.197005 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" Oct 30 19:31:11.197228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.197045 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9" event={"ID":"ca787e75-dcf6-4450-9725-406d89f8dd6f","Type":"ContainerDied","Data":"e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81"} Oct 30 19:31:11.197228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.197100 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e78f5a306083a2ce97b09f4b1ed0a5bdbf7163745ceea7667612f98370286d81" Oct 30 19:31:11.209008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:11.208985 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:12.548174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:12.548138 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:31:13.078646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.078612 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:31:13.079593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.079568 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ca787e75-dcf6-4450-9725-406d89f8dd6f" containerName="revoke-keys" Oct 30 19:31:13.079717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.079594 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca787e75-dcf6-4450-9725-406d89f8dd6f" containerName="revoke-keys" Oct 30 19:31:13.079784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.079752 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca787e75-dcf6-4450-9725-406d89f8dd6f" containerName="revoke-keys" Oct 30 19:31:13.083880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.083859 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.100692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.100668 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:31:13.189578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.189543 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.189753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.189678 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.189796 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.189749 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.189876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.189852 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6sbw\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.189951 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.189939 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.190020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.190008 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.190055 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.190037 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.190129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.190119 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.190164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.190146 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291430 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291561 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291593 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6sbw\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291671 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291826 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291874 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291922 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.291972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.291948 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.292072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.292000 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.292072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.292026 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.292216 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.292191 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.292282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.292257 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.292645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.292624 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.293141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.293115 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.294003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.293982 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.294340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.294321 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.300578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.300553 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.300668 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.300589 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6sbw\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\") pod \"e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.360998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.360911 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs"] Oct 30 19:31:13.366842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.366821 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.371465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.371374 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:31:13.372238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.372198 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:13.376094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.376064 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs"] Oct 30 19:31:13.433074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.432753 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:13.634514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.631886 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.634514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.631964 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.634514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.632137 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85fpt\" (UniqueName: \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.742516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.734489 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85fpt\" (UniqueName: \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.742516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.741665 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.742516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.741756 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.745063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.745026 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.745802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.745775 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.781534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.780994 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:31:13.804776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.804742 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:31:13.805165 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.805143 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843072 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843140 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrmb\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843226 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843297 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843328 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843362 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843388 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843526 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.843855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.843616 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.919504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.918464 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85fpt\" (UniqueName: \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\") pod \"maas-api-revoke-sub-34e1be6f-7khrs\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:13.944560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.944521 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:31:13.950180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950131 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrmb\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950243 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950315 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950344 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950378 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950406 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950529 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950616 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.950770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.950674 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.951682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.951651 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.952181 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.952153 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.953587 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.953567 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.956552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.953770 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:13.956694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.954710 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:31:13.956839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.955838 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:13.957687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.957665 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.015005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:13.981018 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.054522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.037050 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.054522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.037464 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.054522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.039184 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrmb\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.129468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.128462 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274336 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274415 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274482 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274527 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274557 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274583 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbng6\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274733 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274796 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.293133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.274969 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.349964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.349885 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:14.424677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.408938 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.424677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.409029 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.424677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.409167 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.424677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.423121 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.409281 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.425426 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.425509 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.425555 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.425587 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.425624 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbng6\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.428498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.428199 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.455643 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.455482 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbng6\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.558518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.558435 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:14.821909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:14.762725 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:15.101226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.101179 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:15.101860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.101809 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:15.259035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.258997 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:15.446170 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:31:15.446079 6907 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.165s" Oct 30 19:31:15.541469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.532145 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:15.574245 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.557916 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\") pod \"e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:15.576720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.576682 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:15.604730 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:15.585988 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8330f71_b86b_46fe_bb16_9e7c979308a8.slice/crio-bc627ab4a3b2d6f2f7ae8b99de3adccae42fdc8ac135cf27511f1462fdcac2a7 WatchSource:0}: Error finding container bc627ab4a3b2d6f2f7ae8b99de3adccae42fdc8ac135cf27511f1462fdcac2a7: Status 404 returned error can't find the container with id bc627ab4a3b2d6f2f7ae8b99de3adccae42fdc8ac135cf27511f1462fdcac2a7 Oct 30 19:31:15.604730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.593562 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:31:15.639547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.637082 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:15.644621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.643104 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:15.644621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.643195 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:15.644621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.643243 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:15.687675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.687630 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:15.890848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.863021 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:15.890848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:15.869818 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:16.197481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.193291 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:16.650525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.618386 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:31:16.650525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.619087 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:16.650525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.620056 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:31:16.705395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.704799 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:16.730623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.705670 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:16.730623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.706555 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:16.730623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.706693 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:31:16.763486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.733893 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs"] Oct 30 19:31:16.876270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.876167 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:16.881993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.881958 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:31:16.893653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:16.893603 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:17.212525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.201619 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:17.212525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.202337 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:17.765727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.662224 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.773756 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.776146 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" event={"ID":"e8330f71-b86b-46fe-bb16-9e7c979308a8","Type":"ContainerStarted","Data":"bc627ab4a3b2d6f2f7ae8b99de3adccae42fdc8ac135cf27511f1462fdcac2a7"} Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.814561 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.776202 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.776879 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 19:31:17.830959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.776913 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:17.954139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:17.954096 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:18.115530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.108685 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 19:31:18.222256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.210456 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:18.543920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.543872 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.556612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.556076 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.556612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.556161 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.566897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.561829 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:31:18.566897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.561895 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:18.566897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.561929 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:31:18.566897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.561961 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:18.566897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.561991 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:31:18.624394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.622521 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:31:18.624394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.622568 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:31:18.787638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.787592 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" event={"ID":"ebefcc59-4d77-49f1-8c01-00792e94fe3b","Type":"ContainerStarted","Data":"3fc0a422b1f78ba1b0193bd7a51ee666d3f2fea45ebe2fdf0dcbc816a40ce4ac"} Oct 30 19:31:18.789680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.789644 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" event={"ID":"e8330f71-b86b-46fe-bb16-9e7c979308a8","Type":"ContainerStarted","Data":"8c5da4c5407caa38ad42dd63f79758c5d36b81d556275e54fdbfb446540ff9d5"} Oct 30 19:31:18.791318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.791290 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" event={"ID":"f2228b87-c3bd-48df-a080-3fb7de832c68","Type":"ContainerStarted","Data":"0d1be0a62fc0ef1344470773e88e709d94b702d05b3a786f1fe5a63395ce52ae"} Oct 30 19:31:18.810845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.810796 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podStartSLOduration=5.810777639 podStartE2EDuration="5.810777639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:18.809194639 +0000 UTC m=+684.785420588" watchObservedRunningTime="2026-10-30 19:31:18.810777639 +0000 UTC m=+684.787003585" Oct 30 19:31:18.924561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.924514 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.924843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.924814 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.925025 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.925000 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:18.960699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:18.960634 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:31:19.436113 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.436077 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:19.436519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.436495 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:19.436691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.436667 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:19.800889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.800849 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" event={"ID":"4987b25d-8d31-4a79-a263-673e137e0bd6","Type":"ContainerStarted","Data":"a7f239150a327ed82aca0b256d2e51e6bdd0c117b42947fd8e23dbe7a2ec51e6"} Oct 30 19:31:19.801080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.800903 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" event={"ID":"4987b25d-8d31-4a79-a263-673e137e0bd6","Type":"ContainerStarted","Data":"9f8d93dc1405e0cb508c39a08f451a4bb3dabe85bd8d90b6f4a2b434e0815f21"} Oct 30 19:31:19.803402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.803369 6907 generic.go:356] "Generic (PLEG): container finished" podID="f2228b87-c3bd-48df-a080-3fb7de832c68" containerID="2ac5e50624446c891defc94864beb99537c332334bd2baed41f1919c18c36d8b" exitCode=0 Oct 30 19:31:19.803644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.803411 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" event={"ID":"f2228b87-c3bd-48df-a080-3fb7de832c68","Type":"ContainerDied","Data":"2ac5e50624446c891defc94864beb99537c332334bd2baed41f1919c18c36d8b"} Oct 30 19:31:19.805403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.805350 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" event={"ID":"ebefcc59-4d77-49f1-8c01-00792e94fe3b","Type":"ContainerStarted","Data":"f9f76ad8151216d095083691c0d3f97bc4d2063d7361590c8aa801dd5e5841f0"} Oct 30 19:31:19.899777 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.897651 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:19.908600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.908542 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podStartSLOduration=6.908521683 podStartE2EDuration="6.908521683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.834838169 +0000 UTC m=+685.811064112" watchObservedRunningTime="2026-10-30 19:31:19.908521683 +0000 UTC m=+685.884747628" Oct 30 19:31:19.997472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:19.978941 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podStartSLOduration=6.978917877 podStartE2EDuration="6.978917877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.947758472 +0000 UTC m=+685.923984417" watchObservedRunningTime="2026-10-30 19:31:19.978917877 +0000 UTC m=+685.955143821" Oct 30 19:31:20.359656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.359600 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:20.371762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.371728 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:20.372084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.372036 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:20.447469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.443910 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:20.447469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.443978 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:20.916619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.916583 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:20.916853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:20.916833 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:21.374898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.369573 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:21.374898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.369654 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:21.420661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.420619 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:21.437471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.434878 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:21.437471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.434942 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:21.518182 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.518144 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\" (UniqueName: \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\") pod \"f2228b87-c3bd-48df-a080-3fb7de832c68\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " Oct 30 19:31:21.518396 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.518337 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\") pod \"f2228b87-c3bd-48df-a080-3fb7de832c68\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " Oct 30 19:31:21.518571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.518549 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\") pod \"f2228b87-c3bd-48df-a080-3fb7de832c68\" (UID: \"f2228b87-c3bd-48df-a080-3fb7de832c68\") " Oct 30 19:31:21.518925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.518902 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca" pod "f2228b87-c3bd-48df-a080-3fb7de832c68" (UID: "f2228b87-c3bd-48df-a080-3fb7de832c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:21.519381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.519363 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:21.521580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.521549 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt" pod "f2228b87-c3bd-48df-a080-3fb7de832c68" (UID: "f2228b87-c3bd-48df-a080-3fb7de832c68"). InnerVolumeSpecName "kube-api-access-85fpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:21.521690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.521607 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca" pod "f2228b87-c3bd-48df-a080-3fb7de832c68" (UID: "f2228b87-c3bd-48df-a080-3fb7de832c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:21.620710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.620608 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2228b87-c3bd-48df-a080-3fb7de832c68-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:21.620710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.620662 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85fpt\" (UniqueName: \"kubernetes.io/projected/f2228b87-c3bd-48df-a080-3fb7de832c68-kube-api-access-85fpt\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:21.918221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.918129 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:21.918582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.918549 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:22.033469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.968518 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" event={"ID":"f2228b87-c3bd-48df-a080-3fb7de832c68","Type":"ContainerDied","Data":"0d1be0a62fc0ef1344470773e88e709d94b702d05b3a786f1fe5a63395ce52ae"} Oct 30 19:31:22.033469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.968628 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1be0a62fc0ef1344470773e88e709d94b702d05b3a786f1fe5a63395ce52ae" Oct 30 19:31:22.033469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:21.968709 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs" Oct 30 19:31:22.412828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.412325 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:22.412828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.412384 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:22.469528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.468606 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:22.469528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.468690 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:22.909412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.909022 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:22.909412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:22.909075 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:23.358467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.357854 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:23.358467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.357916 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:23.459505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.458991 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:23.473477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.463324 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:23.907469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.906604 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:23.907469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:23.906690 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:24.361470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.360897 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:24.361470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.360966 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:24.444469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.434924 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:24.444469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.435004 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:24.901828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.901361 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:24.901828 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:24.901423 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:25.396488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.367824 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:25.396488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.367907 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:25.436470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.434801 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:31:25.436470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.434861 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 19:31:25.917147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.917108 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:25.926800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:25.917815 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:26.359108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.359066 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 19:31:26.359276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.359147 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 19:31:26.891033 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.890992 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:31:26.891253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.891058 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 19:31:26.968157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.968124 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:26.969183 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.969150 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:26.970259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:26.970237 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:31:27.236529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.236491 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b"] Oct 30 19:31:27.237691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.237667 6907 cpu_manager.go:414] "containerMap: removing container" podUID="f2228b87-c3bd-48df-a080-3fb7de832c68" containerName="revoke-keys" Oct 30 19:31:27.237761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.237696 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2228b87-c3bd-48df-a080-3fb7de832c68" containerName="revoke-keys" Oct 30 19:31:27.237963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.237947 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2228b87-c3bd-48df-a080-3fb7de832c68" containerName="revoke-keys" Oct 30 19:31:27.247694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.247663 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b"] Oct 30 19:31:27.248402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.248035 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.255095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.255056 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:27.255432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.255413 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:31:27.379793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.379750 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqb2t\" (UniqueName: \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.380011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.379824 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.380136 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.380110 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.481729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.481686 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqb2t\" (UniqueName: \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.481906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.481744 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.482085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.482058 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.482741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.482712 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.485259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.485232 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.492759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.492695 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqb2t\" (UniqueName: \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\") pod \"maas-api-revoke-sub-3dc79d9c-ntx9b\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.570046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.570007 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:27.600918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.599845 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:27.600918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.600897 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:27.602838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.602240 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:31:27.896202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.896170 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:27.896602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.896582 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:27.898629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.898607 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:31:27.991971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:27.991924 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b"] Oct 30 19:31:28.000157 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:28.000126 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e7a0d9_0208_4efa_b2fc_d30df5899c46.slice/crio-198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4 WatchSource:0}: Error finding container 198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4: Status 404 returned error can't find the container with id 198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4 Oct 30 19:31:28.456688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.456650 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" event={"ID":"46e7a0d9-0208-4efa-b2fc-d30df5899c46","Type":"ContainerStarted","Data":"daabf3c0bdde098dc6c5f19845caddaeb62bdac4e71f60c0aec52b245ef0c0cf"} Oct 30 19:31:28.456897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.456716 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" event={"ID":"46e7a0d9-0208-4efa-b2fc-d30df5899c46","Type":"ContainerStarted","Data":"198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4"} Oct 30 19:31:28.561786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.561753 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:31:28.573249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.573142 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647370 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647422 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647483 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647513 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647594 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647645 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647687 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x76p\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647716 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.647873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.647757 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.662092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.662031 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" podStartSLOduration=1.662010872 podStartE2EDuration="1.662010872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:28.505693484 +0000 UTC m=+694.481919430" watchObservedRunningTime="2026-10-30 19:31:28.662010872 +0000 UTC m=+694.638236817" Oct 30 19:31:28.674000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.673967 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:31:28.682558 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.682532 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:31:28.688523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.688500 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:31:28.688853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.688634 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.699386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.699359 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:31:28.704489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.704462 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.715545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.715517 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:31:28.752625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.752587 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.752866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.752842 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753032 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753160 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753389 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753561 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753728 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x76p\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.753871 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753858 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.754006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.753991 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.754193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.754167 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.755858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.755835 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.756263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.756242 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.757224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.757195 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.757531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.757507 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.758016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.757993 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.760807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.760785 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.774966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.774706 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.776668 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.776639 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x76p\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\") pod \"e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:28.855757 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855718 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.855938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855761 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.855938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855784 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.855938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855880 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.855938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855918 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.855992 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccbj\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856025 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856166 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhsr\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856229 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856215 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856415 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856384 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856484 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856508 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856531 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856669 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856708 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856730 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.856795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856772 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.856795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.856789 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.958644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958615 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.958794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958692 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.958794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958708 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.958794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958735 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.958971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958814 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.958971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.958858 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959037 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959098 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959106 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959125 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959191 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959256 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959282 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959310 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959385 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959429 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959477 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccbj\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959517 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959520 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959384 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959645 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959657 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhsr\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959666 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959657 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.959765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959689 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.960062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959884 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.960062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.959913 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.960307 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.960283 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.961925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.961902 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.962228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.962205 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.962323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.962311 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.962475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.962438 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.968262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.968237 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.970962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.970943 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.973007 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.972983 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhsr\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\") pod \"e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:28.973087 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.973078 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccbj\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:28.973304 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:28.973292 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:29.007247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.007215 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:29.048555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.047873 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:29.273910 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:29.273880 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cba290_51b0_46a3_8dd0_0472b0aab59e.slice/crio-18d661a6ce5ea461ebcb9f6827e484ea804d724d3218ba765aa0e01ac6414ca7 WatchSource:0}: Error finding container 18d661a6ce5ea461ebcb9f6827e484ea804d724d3218ba765aa0e01ac6414ca7: Status 404 returned error can't find the container with id 18d661a6ce5ea461ebcb9f6827e484ea804d724d3218ba765aa0e01ac6414ca7 Oct 30 19:31:29.274043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.273966 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:31:29.276560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.276523 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.276662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.276606 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.276662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.276642 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.311503 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:29.311470 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342c0ddb_5be4_46be_8ee5_704bad5babf8.slice/crio-3648de871e20296ae598718123895435278e097a2079e7b1046619d2958c2eb7 WatchSource:0}: Error finding container 3648de871e20296ae598718123895435278e097a2079e7b1046619d2958c2eb7: Status 404 returned error can't find the container with id 3648de871e20296ae598718123895435278e097a2079e7b1046619d2958c2eb7 Oct 30 19:31:29.311661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.311640 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:31:29.314412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.314375 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.314512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.314478 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.314551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.314526 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.382340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.382265 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:31:29.383919 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:29.383889 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81102270_6923_4e17_8da4_4b3829a3249b.slice/crio-b4ace794d67a2dd41a8cdfad690f1e50fc105647017efd5ddd08b99daabf06a7 WatchSource:0}: Error finding container b4ace794d67a2dd41a8cdfad690f1e50fc105647017efd5ddd08b99daabf06a7: Status 404 returned error can't find the container with id b4ace794d67a2dd41a8cdfad690f1e50fc105647017efd5ddd08b99daabf06a7 Oct 30 19:31:29.386956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.386921 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.387069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.386999 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.387069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.387037 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:29.481973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.481938 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" event={"ID":"342c0ddb-5be4-46be-8ee5-704bad5babf8","Type":"ContainerStarted","Data":"5ce2193af368ac329e6f5da48be3459969f51da391e5380837bffdf6b6df62c2"} Oct 30 19:31:29.482097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.482003 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" event={"ID":"342c0ddb-5be4-46be-8ee5-704bad5babf8","Type":"ContainerStarted","Data":"3648de871e20296ae598718123895435278e097a2079e7b1046619d2958c2eb7"} Oct 30 19:31:29.484044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.484019 6907 generic.go:356] "Generic (PLEG): container finished" podID="46e7a0d9-0208-4efa-b2fc-d30df5899c46" containerID="daabf3c0bdde098dc6c5f19845caddaeb62bdac4e71f60c0aec52b245ef0c0cf" exitCode=0 Oct 30 19:31:29.484180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.484168 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" event={"ID":"46e7a0d9-0208-4efa-b2fc-d30df5899c46","Type":"ContainerDied","Data":"daabf3c0bdde098dc6c5f19845caddaeb62bdac4e71f60c0aec52b245ef0c0cf"} Oct 30 19:31:29.486963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.486881 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" event={"ID":"b2cba290-51b0-46a3-8dd0-0472b0aab59e","Type":"ContainerStarted","Data":"b7fb7f26f2811b1c7dda37e1b4796090293b42051551224cd3cc4c8f1c99aa49"} Oct 30 19:31:29.486963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.486928 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" event={"ID":"b2cba290-51b0-46a3-8dd0-0472b0aab59e","Type":"ContainerStarted","Data":"18d661a6ce5ea461ebcb9f6827e484ea804d724d3218ba765aa0e01ac6414ca7"} Oct 30 19:31:29.491668 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.490524 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" event={"ID":"81102270-6923-4e17-8da4-4b3829a3249b","Type":"ContainerStarted","Data":"b4ace794d67a2dd41a8cdfad690f1e50fc105647017efd5ddd08b99daabf06a7"} Oct 30 19:31:29.523517 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.516032 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podStartSLOduration=1.516017059 podStartE2EDuration="1.516017059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:29.504279229 +0000 UTC m=+695.480505175" watchObservedRunningTime="2026-10-30 19:31:29.516017059 +0000 UTC m=+695.492243080" Oct 30 19:31:29.658477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.651488 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podStartSLOduration=1.651466908 podStartE2EDuration="1.651466908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:29.61685457 +0000 UTC m=+695.593080515" watchObservedRunningTime="2026-10-30 19:31:29.651466908 +0000 UTC m=+695.627692856" Oct 30 19:31:29.681488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.679899 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podStartSLOduration=1.679879139 podStartE2EDuration="1.679879139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:29.667519754 +0000 UTC m=+695.643745699" watchObservedRunningTime="2026-10-30 19:31:29.679879139 +0000 UTC m=+695.656105083" Oct 30 19:31:29.974578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:29.974528 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:30.007730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.007689 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:30.009336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.009312 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:30.009400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.009362 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:30.050465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.050114 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:30.050465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.050280 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:30.050465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.050326 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:30.515745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:30.515715 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" event={"ID":"81102270-6923-4e17-8da4-4b3829a3249b","Type":"ContainerStarted","Data":"67d3879e9d7f74ed76825b86e42f695d2bd829a642ec46fe5cbf503adc9f517b"} Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.090845 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.090903 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.091501 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.091545 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.091879 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:31.106510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.091917 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:31.226392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.226367 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:31.315294 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.315264 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\") pod \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " Oct 30 19:31:31.315702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.315662 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\" (UniqueName: \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\") pod \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " Oct 30 19:31:31.316314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.316298 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\") pod \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\" (UID: \"46e7a0d9-0208-4efa-b2fc-d30df5899c46\") " Oct 30 19:31:31.316753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.316729 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca" pod "46e7a0d9-0208-4efa-b2fc-d30df5899c46" (UID: "46e7a0d9-0208-4efa-b2fc-d30df5899c46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:31.318398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.318233 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.319917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.319855 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca" pod "46e7a0d9-0208-4efa-b2fc-d30df5899c46" (UID: "46e7a0d9-0208-4efa-b2fc-d30df5899c46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:31.319917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.319888 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t" pod "46e7a0d9-0208-4efa-b2fc-d30df5899c46" (UID: "46e7a0d9-0208-4efa-b2fc-d30df5899c46"). InnerVolumeSpecName "kube-api-access-sqb2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:31.416470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.415534 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm"] Oct 30 19:31:31.422786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.416686 6907 cpu_manager.go:414] "containerMap: removing container" podUID="46e7a0d9-0208-4efa-b2fc-d30df5899c46" containerName="revoke-keys" Oct 30 19:31:31.422786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.416713 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e7a0d9-0208-4efa-b2fc-d30df5899c46" containerName="revoke-keys" Oct 30 19:31:31.422786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.416975 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e7a0d9-0208-4efa-b2fc-d30df5899c46" containerName="revoke-keys" Oct 30 19:31:31.425928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.425830 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46e7a0d9-0208-4efa-b2fc-d30df5899c46-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.426096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.426078 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqb2t\" (UniqueName: \"kubernetes.io/projected/46e7a0d9-0208-4efa-b2fc-d30df5899c46-kube-api-access-sqb2t\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:31.438472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.430635 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm"] Oct 30 19:31:31.470553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.470516 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.623865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.623801 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.624478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.624434 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9k5\" (UniqueName: \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.624800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.624778 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.731561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.730489 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.731561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.730570 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9k5\" (UniqueName: \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.731561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.730729 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.775485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.767614 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.844468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.777690 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" Oct 30 19:31:31.844468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.783669 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:31.844468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.823923 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b" event={"ID":"46e7a0d9-0208-4efa-b2fc-d30df5899c46","Type":"ContainerDied","Data":"198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4"} Oct 30 19:31:31.844468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:31.824024 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="198ed67cdf84224486f165d2a2c4805e2cc4a5f1d1925e435a61ea67b55db6f4" Oct 30 19:31:32.017465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.016639 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.017465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.016750 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:32.019232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.018029 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9k5\" (UniqueName: \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\") pod \"maas-api-revoke-sub-962d6fe8-lp6gm\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:32.019232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.018493 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.019232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.018545 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:32.055469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.053581 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:32.055469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.053652 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:32.219475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.218795 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:32.910592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:32.909403 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm"] Oct 30 19:31:33.031534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.030217 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.031534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.030391 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:33.068497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.065659 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.068497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.065733 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:33.068497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.065937 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.068497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.065973 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:33.120986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.120251 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" event={"ID":"e48d1ec3-959e-4461-ab7e-d648a38119a8","Type":"ContainerStarted","Data":"d4a373c73b9a8b31412300b8d6e7e5361aaff75308c784a2e2cd13ccaaac55ad"} Oct 30 19:31:33.975180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.975143 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:33.975804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:33.975771 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:34.029547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.029490 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.029849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.029801 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:34.055678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.055638 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.056178 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.056143 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:34.207469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.207410 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" event={"ID":"e48d1ec3-959e-4461-ab7e-d648a38119a8","Type":"ContainerStarted","Data":"4871f46ebbbdeb77fd607641fcdfabe486de7a78cc7f8909a08e35985db5c7fc"} Oct 30 19:31:34.314419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.314183 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" podStartSLOduration=3.314161493 podStartE2EDuration="3.314161493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:34.23176855 +0000 UTC m=+700.207994497" watchObservedRunningTime="2026-10-30 19:31:34.314161493 +0000 UTC m=+700.290387438" Oct 30 19:31:34.991468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.985838 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:34.991468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:34.985917 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:35.025583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.024555 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:35.025583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.024616 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:35.051471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.050855 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:35.051471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.050917 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:35.223537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.223501 6907 generic.go:356] "Generic (PLEG): container finished" podID="e48d1ec3-959e-4461-ab7e-d648a38119a8" containerID="4871f46ebbbdeb77fd607641fcdfabe486de7a78cc7f8909a08e35985db5c7fc" exitCode=0 Oct 30 19:31:35.223712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:35.223594 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" event={"ID":"e48d1ec3-959e-4461-ab7e-d648a38119a8","Type":"ContainerDied","Data":"4871f46ebbbdeb77fd607641fcdfabe486de7a78cc7f8909a08e35985db5c7fc"} Oct 30 19:31:36.006511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.005187 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.006511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.005255 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:36.010532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.010497 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.010800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.010768 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:36.079377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.079341 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.082140 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.082102 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:36.538316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.538282 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:36.729343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.729306 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\" (UniqueName: \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\") pod \"e48d1ec3-959e-4461-ab7e-d648a38119a8\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " Oct 30 19:31:36.729594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.729371 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\") pod \"e48d1ec3-959e-4461-ab7e-d648a38119a8\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " Oct 30 19:31:36.729678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.729644 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\") pod \"e48d1ec3-959e-4461-ab7e-d648a38119a8\" (UID: \"e48d1ec3-959e-4461-ab7e-d648a38119a8\") " Oct 30 19:31:36.729839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.729815 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca" pod "e48d1ec3-959e-4461-ab7e-d648a38119a8" (UID: "e48d1ec3-959e-4461-ab7e-d648a38119a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:36.730460 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.730415 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:36.731832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.731801 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5" pod "e48d1ec3-959e-4461-ab7e-d648a38119a8" (UID: "e48d1ec3-959e-4461-ab7e-d648a38119a8"). InnerVolumeSpecName "kube-api-access-4m9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:36.732197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.732180 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca" pod "e48d1ec3-959e-4461-ab7e-d648a38119a8" (UID: "e48d1ec3-959e-4461-ab7e-d648a38119a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:36.832913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.832872 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e48d1ec3-959e-4461-ab7e-d648a38119a8-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:36.832913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.832913 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m9k5\" (UniqueName: \"kubernetes.io/projected/e48d1ec3-959e-4461-ab7e-d648a38119a8-kube-api-access-4m9k5\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:36.974906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.974394 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 19:31:36.974906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:36.974475 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 19:31:37.008179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.008077 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:31:37.008179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.008146 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 19:31:37.062482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.050713 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:31:37.062482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.050775 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 19:31:37.468549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.452557 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" event={"ID":"e48d1ec3-959e-4461-ab7e-d648a38119a8","Type":"ContainerDied","Data":"d4a373c73b9a8b31412300b8d6e7e5361aaff75308c784a2e2cd13ccaaac55ad"} Oct 30 19:31:37.468549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.452736 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4a373c73b9a8b31412300b8d6e7e5361aaff75308c784a2e2cd13ccaaac55ad" Oct 30 19:31:37.468549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:37.456846 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm" Oct 30 19:31:37.788690 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:31:37.788615 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:31:38.019927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.019882 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:38.022595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.022571 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:38.025761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.025729 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:38.026977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.026792 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:31:38.027343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.027201 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:38.028935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.028854 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:31:38.237432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.229108 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:38.239414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.239388 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:38.247863 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:38.246631 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:31:41.706465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.702871 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w"] Oct 30 19:31:41.706465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.704120 6907 cpu_manager.go:414] "containerMap: removing container" podUID="e48d1ec3-959e-4461-ab7e-d648a38119a8" containerName="revoke-keys" Oct 30 19:31:41.706465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.704154 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48d1ec3-959e-4461-ab7e-d648a38119a8" containerName="revoke-keys" Oct 30 19:31:41.706465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.704397 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48d1ec3-959e-4461-ab7e-d648a38119a8" containerName="revoke-keys" Oct 30 19:31:41.710484 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.708547 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.730636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.728775 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:41.730636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.728927 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:31:41.730636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.729363 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w"] Oct 30 19:31:41.786066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.786027 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.786294 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.786274 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.786406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.786389 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.888172 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.888133 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.888364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.888349 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.888486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.888467 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.888987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.888960 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.890778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.890756 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:41.903149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:41.903123 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") pod \"maas-api-revoke-sub-34097152-jxq2w\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:42.046252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:42.046220 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:42.336346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:42.336309 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w"] Oct 30 19:31:42.336663 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:42.336632 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39133361_18df_4262_9e9b_93396062c1d3.slice/crio-ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b WatchSource:0}: Error finding container ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b: Status 404 returned error can't find the container with id ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b Oct 30 19:31:42.633013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:42.632971 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" event={"ID":"39133361-18df-4262-9e9b-93396062c1d3","Type":"ContainerStarted","Data":"ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b"} Oct 30 19:31:43.663806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.663727 6907 generic.go:356] "Generic (PLEG): container finished" podID="39133361-18df-4262-9e9b-93396062c1d3" containerID="8c3e2d1e4e94796e4284c32817d56d90e99771034b8fd420a26b6e90acf7c02b" exitCode=0 Oct 30 19:31:43.664340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.664230 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" event={"ID":"39133361-18df-4262-9e9b-93396062c1d3","Type":"ContainerDied","Data":"8c3e2d1e4e94796e4284c32817d56d90e99771034b8fd420a26b6e90acf7c02b"} Oct 30 19:31:43.737193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.737158 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:31:43.741685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.741664 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.758004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.757976 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:31:43.767477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.767418 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.767717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.767696 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzms\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.767855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.767839 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.767988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.767964 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.768119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.768008 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.768119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.768069 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.768119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.768102 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.768419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.768169 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.768535 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.768414 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870410 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870539 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870570 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzms\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870593 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870718 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870767 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870836 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870873 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.870989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.870943 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.871050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.871023 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.871106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.871059 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.871374 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.871351 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.871515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.871421 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.871604 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.871586 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.872755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.872739 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.873244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.873224 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.879695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.879673 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzms\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:43.880046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:43.880026 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\") pod \"e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:44.062299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.062266 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:44.357610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.357577 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:31:44.358854 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:44.358826 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44654834_b2a2_41ca_b840_89809c0fea2e.slice/crio-1f1c87c16262443b32b5401007839c5670059e115afbd684624f4b30f1ae8e1f WatchSource:0}: Error finding container 1f1c87c16262443b32b5401007839c5670059e115afbd684624f4b30f1ae8e1f: Status 404 returned error can't find the container with id 1f1c87c16262443b32b5401007839c5670059e115afbd684624f4b30f1ae8e1f Oct 30 19:31:44.362507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.362470 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:44.362628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.362557 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:44.362628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.362601 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:44.819499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.819090 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" event={"ID":"44654834-b2a2-41ca-b840-89809c0fea2e","Type":"ContainerStarted","Data":"1794fca089134a49d49f4e820ca6a9d99625e938764d2a1fd346ca5447c0d085"} Oct 30 19:31:44.819499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.819182 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" event={"ID":"44654834-b2a2-41ca-b840-89809c0fea2e","Type":"ContainerStarted","Data":"1f1c87c16262443b32b5401007839c5670059e115afbd684624f4b30f1ae8e1f"} Oct 30 19:31:44.922070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:44.922005 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podStartSLOduration=1.9219800889999998 podStartE2EDuration="1.921980089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:44.86009518 +0000 UTC m=+710.836321156" watchObservedRunningTime="2026-10-30 19:31:44.921980089 +0000 UTC m=+710.898206034" Oct 30 19:31:45.070015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.069286 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:45.070015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.069421 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:45.070015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.069547 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:45.825070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.825024 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:45.828164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.828124 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:45.999038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:45.998992 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:46.078536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.066804 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:46.115472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.081692 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:46.169407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.169354 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:46.506960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.506588 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:46.628233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.609921 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:46.833467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.832649 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:46.833467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.833241 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:46.834031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.833614 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 19:31:46.881782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.868937 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:31:46.881782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:46.869011 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:31:47.110627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.101140 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.136563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.126790 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:31:47.170593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.166623 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:47.170593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.166690 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:47.575841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.541080 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.575841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.543044 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:31:47.636388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.626526 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:31:47.643400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.643371 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:47.739967 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.694664 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.945658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.851539 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.945658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.852399 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:47.945658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.859903 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:47.945658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:47.861833 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:48.059112 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.038563 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.077520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.073392 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:48.077520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.073474 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:48.106342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.106310 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:48.107030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.106996 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:48.270502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.264021 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") pod \"39133361-18df-4262-9e9b-93396062c1d3\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " Oct 30 19:31:48.270502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.264213 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\") pod \"39133361-18df-4262-9e9b-93396062c1d3\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " Oct 30 19:31:48.270502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.264528 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\") pod \"39133361-18df-4262-9e9b-93396062c1d3\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " Oct 30 19:31:48.437881 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.437841 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca" pod "39133361-18df-4262-9e9b-93396062c1d3" (UID: "39133361-18df-4262-9e9b-93396062c1d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:48.564522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.552226 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:48.675609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.580675 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39133361-18df-4262-9e9b-93396062c1d3-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:48.693509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.675921 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.759620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.754801 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:48.879644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.860232 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.217947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:48.864590 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.222508 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.013630 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:31:49.234108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.054528 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:31:49.236009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.071065 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:49.236227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.236200 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:49.253963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.132872 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.273059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.258612 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:49.273347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.273316 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.387338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.387301 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca" pod "39133361-18df-4262-9e9b-93396062c1d3" (UID: "39133361-18df-4262-9e9b-93396062c1d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:49.564471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.551844 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 19:31:49.565889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.565840 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39133361-18df-4262-9e9b-93396062c1d3-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:49.598651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.566708 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.680716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.680658 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" event={"ID":"39133361-18df-4262-9e9b-93396062c1d3","Type":"ContainerDied","Data":"ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b"} Oct 30 19:31:49.685518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.681544 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3f37a1da01b4204dd0ac81e886d9f092ad7b732b2ba8987e165652d510748b" Oct 30 19:31:49.699593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.699539 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w" Oct 30 19:31:49.713654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.713595 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:49.764471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.756680 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.023946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.916873 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh" pod "39133361-18df-4262-9e9b-93396062c1d3" (UID: "39133361-18df-4262-9e9b-93396062c1d3"). InnerVolumeSpecName "kube-api-access-fjkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:50.023946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.918341 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") pod \"39133361-18df-4262-9e9b-93396062c1d3\" (UID: \"39133361-18df-4262-9e9b-93396062c1d3\") " Oct 30 19:31:50.023946 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:49.918613 6907 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/39133361-18df-4262-9e9b-93396062c1d3/volumes/kubernetes.io~projected/kube-api-access-fjkjh Oct 30 19:31:50.023946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.918661 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh" pod "39133361-18df-4262-9e9b-93396062c1d3" (UID: "39133361-18df-4262-9e9b-93396062c1d3"). InnerVolumeSpecName "kube-api-access-fjkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:50.023946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:49.920217 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkjh\" (UniqueName: \"kubernetes.io/projected/39133361-18df-4262-9e9b-93396062c1d3-kube-api-access-fjkjh\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:50.090897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.090765 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 30 19:31:50.118437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.118399 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:50.118851 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.118715 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:50.220646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.220599 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.573534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.231565 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.573534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.266496 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.573534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.274142 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:31:50.654102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.650203 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:50.721776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.721711 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:50.781363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.781316 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:31:50.932531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:50.925022 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 19:31:51.096091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.096053 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:51.096377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.096343 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:51.424598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.424549 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.426622 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.426590 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:51.427371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.427346 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.427832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.427798 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:51.658801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.658705 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:51.736531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.724541 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.785609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.785564 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:51.822471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.818704 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:51.924514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:51.810548 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.143528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.139942 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.143699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.143616 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:52.143699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.143669 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:52.356500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.350713 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-k2kb4" podUID="cfb8121c-04df-44f7-a66e-59df415cf0f7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.28:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.402659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.402556 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:52.402935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.402902 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.424655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.424554 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:52.425115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.424679 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.426133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.426099 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.427060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.427031 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:52.427926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.427901 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.431528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.431501 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.510690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.508416 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:52.510690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.508493 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.680560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.659371 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.681853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.681523 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:52.681853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.681585 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.722692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.722639 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:52.723265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.723233 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:52.728938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.728886 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:31:52.976969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:52.957958 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:53.006881 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.006844 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:53.007209 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.007164 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:53.067507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.065868 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:53.067507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.065934 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:53.436053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.435943 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:53.437092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.436641 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:53.438253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.436681 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:31:53.473841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.436704 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:53.690187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.666717 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:31:53.740620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:53.740567 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.002636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.002569 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.066348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.066313 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:54.066622 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.066590 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:54.124410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.124374 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:31:54.124709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.124681 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.485470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.475214 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:31:54.486262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.478742 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 19:31:54.516465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.478776 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:54.582529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.575772 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:31:54.680072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.680014 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.744765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.744656 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:54.926429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:54.925375 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:55.003158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.003065 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:55.064152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.064117 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:55.064553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.064429 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:55.479126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.479094 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:31:55.491177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.491041 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:31:55.572812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.571578 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:31:55.575919 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.575883 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:31:55.582946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.582920 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:31:55.589413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.586995 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:31:55.589686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.589671 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:31:55.605588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.605536 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg"] Oct 30 19:31:55.607147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.607084 6907 cpu_manager.go:414] "containerMap: removing container" podUID="39133361-18df-4262-9e9b-93396062c1d3" containerName="revoke-keys" Oct 30 19:31:55.607147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.607118 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39133361-18df-4262-9e9b-93396062c1d3" containerName="revoke-keys" Oct 30 19:31:55.607576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.607558 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="39133361-18df-4262-9e9b-93396062c1d3" containerName="revoke-keys" Oct 30 19:31:55.615142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.615117 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:31:55.615271 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.615171 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg"] Oct 30 19:31:55.615271 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.615236 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:31:55.615518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.615500 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.624791 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.624769 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:31:55.687395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.686592 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4ps\" (UniqueName: \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.687395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.686711 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.687395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.686934 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.723489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.720121 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:31:55.789162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.789057 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.789338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.789261 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4ps\" (UniqueName: \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.789410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.789367 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.789802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.789775 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.793559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.793532 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.806456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.806417 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4ps\" (UniqueName: \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\") pod \"maas-api-revoke-sub-89a9b580-7mmhg\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:55.983481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:55.983377 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:56.064570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:56.064138 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:56.064570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:56.064217 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:56.375267 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:56.375240 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f71dcd_6168_411f_8058_b2b0d2d5d774.slice/crio-b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e WatchSource:0}: Error finding container b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e: Status 404 returned error can't find the container with id b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e Oct 30 19:31:56.375732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:56.375709 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg"] Oct 30 19:31:56.781193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:56.781152 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" event={"ID":"61f71dcd-6168-411f-8058-b2b0d2d5d774","Type":"ContainerStarted","Data":"b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e"} Oct 30 19:31:57.064132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:57.064040 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:57.064132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:57.064097 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:57.791044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:57.791010 6907 generic.go:356] "Generic (PLEG): container finished" podID="61f71dcd-6168-411f-8058-b2b0d2d5d774" containerID="7fa92c72597ab05ebc76509e752b46a2857cec7c4a685857f234e025ed90b00e" exitCode=0 Oct 30 19:31:57.791432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:57.791086 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" event={"ID":"61f71dcd-6168-411f-8058-b2b0d2d5d774","Type":"ContainerDied","Data":"7fa92c72597ab05ebc76509e752b46a2857cec7c4a685857f234e025ed90b00e"} Oct 30 19:31:58.063246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.063208 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:58.063427 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.063270 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:58.175125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.175087 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:31:58.180685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.180658 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.199123 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.199083 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:31:58.272636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.272598 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.272838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.272685 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.272838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.272813 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.272958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.272869 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.273010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.272969 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.273063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.273018 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.273174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.273140 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.273255 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.273239 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.273370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.273352 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcxf\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376254 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376381 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcxf\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376569 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376604 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376669 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376718 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376777 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376821 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.376871 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.377400 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.378192 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.378692 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.378928 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.391648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.379529 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.397177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.395500 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.431562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.424682 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcxf\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.433076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.425060 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.447892 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.447855 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\") pod \"e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.509789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.509747 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:31:58.916490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.916437 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:31:58.917403 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:31:58.917374 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9232aebc_4241_415f_90d5_822695de5791.slice/crio-6257f60bfab48d2f67dc980549044a34c616c49015a2b3d5475cb55d77f793fe WatchSource:0}: Error finding container 6257f60bfab48d2f67dc980549044a34c616c49015a2b3d5475cb55d77f793fe: Status 404 returned error can't find the container with id 6257f60bfab48d2f67dc980549044a34c616c49015a2b3d5475cb55d77f793fe Oct 30 19:31:58.920253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.920220 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:58.920370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.920290 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:58.920370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.920319 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:31:58.991145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:58.991122 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:59.005513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.005483 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\") pod \"61f71dcd-6168-411f-8058-b2b0d2d5d774\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " Oct 30 19:31:59.005641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.005560 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\" (UniqueName: \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\") pod \"61f71dcd-6168-411f-8058-b2b0d2d5d774\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " Oct 30 19:31:59.005730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.005713 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\") pod \"61f71dcd-6168-411f-8058-b2b0d2d5d774\" (UID: \"61f71dcd-6168-411f-8058-b2b0d2d5d774\") " Oct 30 19:31:59.005804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.005777 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca" pod "61f71dcd-6168-411f-8058-b2b0d2d5d774" (UID: "61f71dcd-6168-411f-8058-b2b0d2d5d774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:59.006487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.006346 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:59.010789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.010760 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps" pod "61f71dcd-6168-411f-8058-b2b0d2d5d774" (UID: "61f71dcd-6168-411f-8058-b2b0d2d5d774"). InnerVolumeSpecName "kube-api-access-7z4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:59.010912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.010887 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca" pod "61f71dcd-6168-411f-8058-b2b0d2d5d774" (UID: "61f71dcd-6168-411f-8058-b2b0d2d5d774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:59.062758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.062715 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:31:59.063043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.063016 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:31:59.063106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.063073 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:31:59.108635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.108598 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4ps\" (UniqueName: \"kubernetes.io/projected/61f71dcd-6168-411f-8058-b2b0d2d5d774-kube-api-access-7z4ps\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:59.108635 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.108634 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f71dcd-6168-411f-8058-b2b0d2d5d774-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:59.812217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.812174 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" event={"ID":"9232aebc-4241-415f-90d5-822695de5791","Type":"ContainerStarted","Data":"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d"} Oct 30 19:31:59.812411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.812237 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" event={"ID":"9232aebc-4241-415f-90d5-822695de5791","Type":"ContainerStarted","Data":"6257f60bfab48d2f67dc980549044a34c616c49015a2b3d5475cb55d77f793fe"} Oct 30 19:31:59.813994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.813971 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" Oct 30 19:31:59.813994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.813971 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg" event={"ID":"61f71dcd-6168-411f-8058-b2b0d2d5d774","Type":"ContainerDied","Data":"b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e"} Oct 30 19:31:59.814153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.814037 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0993ac16f868d238f9b068a0c6bd95c75c42f52d6991cf179281f7a1dc1df3e" Oct 30 19:31:59.834854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:31:59.833587 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podStartSLOduration=1.833567876 podStartE2EDuration="1.833567876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:59.832909955 +0000 UTC m=+725.809135900" watchObservedRunningTime="2026-10-30 19:31:59.833567876 +0000 UTC m=+725.809793820" Oct 30 19:32:00.063058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:00.062970 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:32:00.063058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:00.063020 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 19:32:00.510520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:00.510480 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:32:00.512331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:00.512302 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:00.512496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:00.512358 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:01.279776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.279735 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:32:01.281064 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.281042 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:32:01.466523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.466483 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96"] Oct 30 19:32:01.467675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.467656 6907 cpu_manager.go:414] "containerMap: removing container" podUID="61f71dcd-6168-411f-8058-b2b0d2d5d774" containerName="revoke-keys" Oct 30 19:32:01.467755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.467677 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f71dcd-6168-411f-8058-b2b0d2d5d774" containerName="revoke-keys" Oct 30 19:32:01.467847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.467828 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f71dcd-6168-411f-8058-b2b0d2d5d774" containerName="revoke-keys" Oct 30 19:32:01.472954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.472926 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.477695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.477639 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:01.478088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.478041 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:01.487056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.487033 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96"] Oct 30 19:32:01.514804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.503610 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.515128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.511615 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:01.515243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.515167 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:01.515366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.515337 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.515430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.515415 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgb9\" (UniqueName: \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.616788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.616688 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.616957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.616927 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.617020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.616980 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgb9\" (UniqueName: \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.617659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.617630 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.619291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.619254 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.625968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.625947 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgb9\" (UniqueName: \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\") pod \"maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:01.790846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:01.790811 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:02.098600 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:02.098570 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75f0f04_9d22_4efc_98a3_f96ebdc41146.slice/crio-91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7 WatchSource:0}: Error finding container 91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7: Status 404 returned error can't find the container with id 91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7 Oct 30 19:32:02.098798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.098773 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96"] Oct 30 19:32:02.515467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.514896 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:02.515467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.514973 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:02.869080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.868977 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" event={"ID":"f75f0f04-9d22-4efc-98a3-f96ebdc41146","Type":"ContainerStarted","Data":"0d234574341e68b06cb3faae49dbeba21f519b29f38587134d92d8c58457149f"} Oct 30 19:32:02.869080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.869044 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" event={"ID":"f75f0f04-9d22-4efc-98a3-f96ebdc41146","Type":"ContainerStarted","Data":"91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7"} Oct 30 19:32:02.889337 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:02.889288 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" podStartSLOduration=1.8892747970000001 podStartE2EDuration="1.889274797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:02.884823933 +0000 UTC m=+728.861049877" watchObservedRunningTime="2026-10-30 19:32:02.889274797 +0000 UTC m=+728.865500742" Oct 30 19:32:03.512810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:03.512712 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:03.513094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:03.513058 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:03.879727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:03.879633 6907 generic.go:356] "Generic (PLEG): container finished" podID="f75f0f04-9d22-4efc-98a3-f96ebdc41146" containerID="0d234574341e68b06cb3faae49dbeba21f519b29f38587134d92d8c58457149f" exitCode=0 Oct 30 19:32:03.880280 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:03.879721 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" event={"ID":"f75f0f04-9d22-4efc-98a3-f96ebdc41146","Type":"ContainerDied","Data":"0d234574341e68b06cb3faae49dbeba21f519b29f38587134d92d8c58457149f"} Oct 30 19:32:04.511372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:04.511331 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:04.511614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:04.511400 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:05.109657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.109625 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:05.216660 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.216581 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\" (UniqueName: \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\") pod \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " Oct 30 19:32:05.216854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.216815 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\") pod \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " Oct 30 19:32:05.216931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.216902 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\") pod \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\" (UID: \"f75f0f04-9d22-4efc-98a3-f96ebdc41146\") " Oct 30 19:32:05.217339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.217308 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca" pod "f75f0f04-9d22-4efc-98a3-f96ebdc41146" (UID: "f75f0f04-9d22-4efc-98a3-f96ebdc41146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:05.218042 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.218005 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:05.219292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.219266 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca" pod "f75f0f04-9d22-4efc-98a3-f96ebdc41146" (UID: "f75f0f04-9d22-4efc-98a3-f96ebdc41146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:05.219583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.219562 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9" pod "f75f0f04-9d22-4efc-98a3-f96ebdc41146" (UID: "f75f0f04-9d22-4efc-98a3-f96ebdc41146"). InnerVolumeSpecName "kube-api-access-hbgb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:05.302290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.302250 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv"] Oct 30 19:32:05.303247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.303224 6907 cpu_manager.go:414] "containerMap: removing container" podUID="f75f0f04-9d22-4efc-98a3-f96ebdc41146" containerName="revoke-keys" Oct 30 19:32:05.303247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.303245 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f75f0f04-9d22-4efc-98a3-f96ebdc41146" containerName="revoke-keys" Oct 30 19:32:05.303435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.303417 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="f75f0f04-9d22-4efc-98a3-f96ebdc41146" containerName="revoke-keys" Oct 30 19:32:05.307337 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.307317 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.316891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.316859 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv"] Oct 30 19:32:05.319661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.319635 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbgb9\" (UniqueName: \"kubernetes.io/projected/f75f0f04-9d22-4efc-98a3-f96ebdc41146-kube-api-access-hbgb9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:05.319748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.319661 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f75f0f04-9d22-4efc-98a3-f96ebdc41146-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:05.420956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.420914 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.421224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.421200 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm822\" (UniqueName: \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.421306 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.421235 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.511367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.511332 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:05.511602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.511401 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:05.523331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.523291 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm822\" (UniqueName: \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.523533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.523334 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.523533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.523501 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.524072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.524053 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.525940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.525922 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.554380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.554340 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm822\" (UniqueName: \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\") pod \"maas-api-revoke-sub-8d7ee3c5-495tv\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.645282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.645239 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:05.957390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.957023 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" event={"ID":"f75f0f04-9d22-4efc-98a3-f96ebdc41146","Type":"ContainerDied","Data":"91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7"} Oct 30 19:32:05.957390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.957122 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91badcd90d330da70fc84f4c4cfa8db914b8ca89fd2cf8602c9bcf9e9857fcc7" Oct 30 19:32:05.957390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.957203 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96" Oct 30 19:32:05.969489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:05.969257 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv"] Oct 30 19:32:05.971491 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:05.971102 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0addc13_c7eb_40b3_9177_f1cc8b94e0d3.slice/crio-cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78 WatchSource:0}: Error finding container cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78: Status 404 returned error can't find the container with id cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78 Oct 30 19:32:06.511280 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:06.511187 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:32:06.511280 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:06.511241 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 19:32:06.969114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:06.969079 6907 generic.go:356] "Generic (PLEG): container finished" podID="a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" containerID="23f089d4a49145a2d2e655e3d090e0ade8097058fc1218cbfb7c28e007b39cd6" exitCode=0 Oct 30 19:32:06.969290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:06.969131 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" event={"ID":"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3","Type":"ContainerDied","Data":"23f089d4a49145a2d2e655e3d090e0ade8097058fc1218cbfb7c28e007b39cd6"} Oct 30 19:32:06.969290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:06.969194 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" event={"ID":"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3","Type":"ContainerStarted","Data":"cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78"} Oct 30 19:32:07.273551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:07.273515 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:32:07.515727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:07.515696 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:32:07.516164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:07.516144 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:32:07.516834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:07.516815 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:32:07.846689 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:07.846098 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:32:08.185751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.185717 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:08.216477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.215121 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\") pod \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " Oct 30 19:32:08.216477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.215320 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\" (UniqueName: \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\") pod \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " Oct 30 19:32:08.216477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.215407 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\") pod \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\" (UID: \"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3\") " Oct 30 19:32:08.216733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.216562 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca" pod "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" (UID: "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:08.227675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.227634 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca" pod "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" (UID: "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:08.228543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.228428 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822" pod "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" (UID: "a0addc13-c7eb-40b3-9177-f1cc8b94e0d3"). InnerVolumeSpecName "kube-api-access-hm822". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:08.318527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.318274 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:08.318527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.318320 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm822\" (UniqueName: \"kubernetes.io/projected/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-kube-api-access-hm822\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:08.318527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:08.318341 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:09.162716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.162678 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" Oct 30 19:32:09.180955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.163389 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv" event={"ID":"a0addc13-c7eb-40b3-9177-f1cc8b94e0d3","Type":"ContainerDied","Data":"cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78"} Oct 30 19:32:09.181247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.181225 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc68f39619a28b7d45027a975434992ab2919bf4e5766b7309fc39192c0eaa78" Oct 30 19:32:09.419528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.418542 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:09.456538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.454633 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:09.570469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.569556 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:09.677631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.677527 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:09.781514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.781470 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:32:09.948528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:09.947540 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": context deadline exceeded" Oct 30 19:32:10.115469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.114611 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:10.224468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.222546 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:10.258468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.242582 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:10.436471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.433602 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:10.436471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.434651 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:32:10.594686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.594569 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:10.612478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.599309 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:32:10.685503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.680588 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:10.685503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.681544 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:10.867884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.788539 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:10.867884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.800039 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 19:32:10.867884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:10.800181 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:11.148721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.147205 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:11.163480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.161692 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:32:11.271173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.234546 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:11.494867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.272628 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:11.525472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.454990 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:11.525472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.495325 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:32:11.544527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.536899 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:32:11.630304 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.630259 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:11.733256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.733206 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:11.796962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:11.796837 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:32:12.101658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.101554 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:12.112688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.112637 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:12.151933 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.151883 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:12.154430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.154392 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:12.497771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.497725 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:12.528780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.520425 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:32:12.600919 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.545966 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw"] Oct 30 19:32:12.629470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.602254 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" containerName="revoke-keys" Oct 30 19:32:12.738525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.738486 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" containerName="revoke-keys" Oct 30 19:32:12.739077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.739052 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" containerName="revoke-keys" Oct 30 19:32:12.789926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.742557 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:12.789926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.558773 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:12.803690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.666742 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:12.879733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:12.807910 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:13.319477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.307542 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:13.454582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.412174 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw"] Oct 30 19:32:13.603712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.603613 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:13.666468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.660530 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:13.744866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.744804 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:13.837391 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.750117 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:13.843659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.843614 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbc7\" (UniqueName: \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:13.844276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.844243 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:13.844723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.776104 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 19:32:13.869811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.849274 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:14.067332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:13.948165 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.071466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.067663 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbc7\" (UniqueName: \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.081477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.072240 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.513773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.081826 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:14.539472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.081868 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.118797 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.118875 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.542466 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.192481 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.280433 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-ddzlh" podUID="0a76f54a-2ca7-401b-86a0-7e51f33fced3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.38:50052\" within 1s: context deadline exceeded" Oct 30 19:32:14.578876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.132705 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:14.624436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.624396 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbc7\" (UniqueName: \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\") pod \"maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:14.754555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.654428 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:32:14.794780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.784086 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:14.795091 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.795040 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:14.799629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.799588 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:14.982921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:14.981768 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:32:15.132979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.132849 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:15.134584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.134560 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:15.188409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.188357 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:15.188870 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.188825 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Oct 30 19:32:15.197260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.189100 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:15.606275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.606231 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:15.650274 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.629027 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:15.889418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.889297 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.991537 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:15.994616 6907 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.8s" Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.994705 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.994750 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.994781 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:32:16.001173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:15.994826 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66"] Oct 30 19:32:16.054954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.054919 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:16.060049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.056187 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.103472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.066760 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:16.287867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.287818 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b28t\" (UniqueName: \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.288083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.287927 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.288236 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.288211 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.481711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.441355 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.482360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.482323 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b28t\" (UniqueName: \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.482596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.482574 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.473761 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.446306 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.485820 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.485846 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66"] Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.485896 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.485918 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:32:16.497468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.485937 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:32:16.549469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.548648 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b28t\" (UniqueName: \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.666034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.665491 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d921ed-wbx66\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.709276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.693786 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:16.786305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.786210 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:16.787129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:16.786753 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:32:17.022467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.022278 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:32:17.214345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.214310 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw"] Oct 30 19:32:17.484171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.483994 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66"] Oct 30 19:32:17.488567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.488539 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" event={"ID":"d336c29f-9a58-4372-b615-225c9eb1fb4c","Type":"ContainerStarted","Data":"f99ec74479acced12ecefabd9b62c8119b945f33863aa2ad2e08b71942224bb5"} Oct 30 19:32:17.490502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.490470 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" event={"ID":"b15e6f73-f603-40e4-83de-a4684d6b87f8","Type":"ContainerStarted","Data":"1728c8a19a621b830217b2e2a7a43dfb7f26c45a308e3340528646679edf40ce"} Oct 30 19:32:17.598415 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.598378 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk"] Oct 30 19:32:17.607729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.607702 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.609646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.609615 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk"] Oct 30 19:32:17.639940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.639904 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcf2\" (UniqueName: \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.640119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.640019 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.640279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.640255 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.741926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.741895 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcf2\" (UniqueName: \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.742375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.742007 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.742375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.742254 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.742853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.742833 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.744348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.744325 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.750501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.750479 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcf2\" (UniqueName: \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\") pod \"maas-api-revoke-sub-c22b0f0a-mggpk\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:17.925707 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:17.925672 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:18.265181 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:18.265150 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e06ff31_d093_4a83_ac9f_13875a3d6e6c.slice/crio-288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc WatchSource:0}: Error finding container 288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc: Status 404 returned error can't find the container with id 288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc Oct 30 19:32:18.265322 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.265290 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk"] Oct 30 19:32:18.502141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.502099 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" event={"ID":"d336c29f-9a58-4372-b615-225c9eb1fb4c","Type":"ContainerStarted","Data":"0d2759ffbc2bd4028f470fa8aa739c948a2278f9886d14c2067543d120f1bea0"} Oct 30 19:32:18.503696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.503669 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" event={"ID":"7e06ff31-d093-4a83-ac9f-13875a3d6e6c","Type":"ContainerStarted","Data":"288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc"} Oct 30 19:32:18.505393 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.505369 6907 generic.go:356] "Generic (PLEG): container finished" podID="b15e6f73-f603-40e4-83de-a4684d6b87f8" containerID="04144c21243c677e397adf2f901f40a9e8d458311a5bd04158c3d2fba2b2874c" exitCode=0 Oct 30 19:32:18.505558 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.505459 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" event={"ID":"b15e6f73-f603-40e4-83de-a4684d6b87f8","Type":"ContainerDied","Data":"04144c21243c677e397adf2f901f40a9e8d458311a5bd04158c3d2fba2b2874c"} Oct 30 19:32:18.523254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:18.523167 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" podStartSLOduration=3.5231534 podStartE2EDuration="3.5231534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:18.517717254 +0000 UTC m=+744.493943209" watchObservedRunningTime="2026-10-30 19:32:18.5231534 +0000 UTC m=+744.499379386" Oct 30 19:32:19.516640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.516547 6907 generic.go:356] "Generic (PLEG): container finished" podID="d336c29f-9a58-4372-b615-225c9eb1fb4c" containerID="0d2759ffbc2bd4028f470fa8aa739c948a2278f9886d14c2067543d120f1bea0" exitCode=0 Oct 30 19:32:19.516640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.516618 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" event={"ID":"d336c29f-9a58-4372-b615-225c9eb1fb4c","Type":"ContainerDied","Data":"0d2759ffbc2bd4028f470fa8aa739c948a2278f9886d14c2067543d120f1bea0"} Oct 30 19:32:19.518403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.518373 6907 generic.go:356] "Generic (PLEG): container finished" podID="7e06ff31-d093-4a83-ac9f-13875a3d6e6c" containerID="48e8356e198c8d90a02b730c18ff5f346c602887cea63d67ec0d9f2deffbd8be" exitCode=0 Oct 30 19:32:19.518557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.518489 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" event={"ID":"7e06ff31-d093-4a83-ac9f-13875a3d6e6c","Type":"ContainerDied","Data":"48e8356e198c8d90a02b730c18ff5f346c602887cea63d67ec0d9f2deffbd8be"} Oct 30 19:32:19.709912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.709884 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:19.793473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.793347 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\") pod \"b15e6f73-f603-40e4-83de-a4684d6b87f8\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " Oct 30 19:32:19.793637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.793506 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\" (UniqueName: \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\") pod \"b15e6f73-f603-40e4-83de-a4684d6b87f8\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " Oct 30 19:32:19.793692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.793668 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\") pod \"b15e6f73-f603-40e4-83de-a4684d6b87f8\" (UID: \"b15e6f73-f603-40e4-83de-a4684d6b87f8\") " Oct 30 19:32:19.793740 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.793721 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca" pod "b15e6f73-f603-40e4-83de-a4684d6b87f8" (UID: "b15e6f73-f603-40e4-83de-a4684d6b87f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:19.794491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.794458 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:19.795711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.795687 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7" pod "b15e6f73-f603-40e4-83de-a4684d6b87f8" (UID: "b15e6f73-f603-40e4-83de-a4684d6b87f8"). InnerVolumeSpecName "kube-api-access-nrbc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:19.795803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.795786 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca" pod "b15e6f73-f603-40e4-83de-a4684d6b87f8" (UID: "b15e6f73-f603-40e4-83de-a4684d6b87f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:19.895579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.895544 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15e6f73-f603-40e4-83de-a4684d6b87f8-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:19.895579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:19.895575 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbc7\" (UniqueName: \"kubernetes.io/projected/b15e6f73-f603-40e4-83de-a4684d6b87f8-kube-api-access-nrbc7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.530825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.530795 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" Oct 30 19:32:20.531259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.530827 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw" event={"ID":"b15e6f73-f603-40e4-83de-a4684d6b87f8","Type":"ContainerDied","Data":"1728c8a19a621b830217b2e2a7a43dfb7f26c45a308e3340528646679edf40ce"} Oct 30 19:32:20.531259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.530882 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1728c8a19a621b830217b2e2a7a43dfb7f26c45a308e3340528646679edf40ce" Oct 30 19:32:20.742722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.742693 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:20.746825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.746801 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:20.814272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814168 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\") pod \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " Oct 30 19:32:20.814272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814251 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\") pod \"d336c29f-9a58-4372-b615-225c9eb1fb4c\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " Oct 30 19:32:20.814594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814332 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\" (UniqueName: \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\") pod \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " Oct 30 19:32:20.814594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814410 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\") pod \"d336c29f-9a58-4372-b615-225c9eb1fb4c\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " Oct 30 19:32:20.814710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814638 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\" (UniqueName: \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\") pod \"d336c29f-9a58-4372-b615-225c9eb1fb4c\" (UID: \"d336c29f-9a58-4372-b615-225c9eb1fb4c\") " Oct 30 19:32:20.814710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814699 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca" pod "d336c29f-9a58-4372-b615-225c9eb1fb4c" (UID: "d336c29f-9a58-4372-b615-225c9eb1fb4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:20.814851 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.814814 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\") pod \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\" (UID: \"7e06ff31-d093-4a83-ac9f-13875a3d6e6c\") " Oct 30 19:32:20.815275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.815242 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca" pod "7e06ff31-d093-4a83-ac9f-13875a3d6e6c" (UID: "7e06ff31-d093-4a83-ac9f-13875a3d6e6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:20.816773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.816745 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2" pod "7e06ff31-d093-4a83-ac9f-13875a3d6e6c" (UID: "7e06ff31-d093-4a83-ac9f-13875a3d6e6c"). InnerVolumeSpecName "kube-api-access-jkcf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:20.816973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.816908 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.816973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.816953 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.817349 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.817323 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca" pod "d336c29f-9a58-4372-b615-225c9eb1fb4c" (UID: "d336c29f-9a58-4372-b615-225c9eb1fb4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:20.817628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.817603 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca" pod "7e06ff31-d093-4a83-ac9f-13875a3d6e6c" (UID: "7e06ff31-d093-4a83-ac9f-13875a3d6e6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:20.817821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.817801 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t" pod "d336c29f-9a58-4372-b615-225c9eb1fb4c" (UID: "d336c29f-9a58-4372-b615-225c9eb1fb4c"). InnerVolumeSpecName "kube-api-access-9b28t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:20.985776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.982112 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.985776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.982143 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcf2\" (UniqueName: \"kubernetes.io/projected/7e06ff31-d093-4a83-ac9f-13875a3d6e6c-kube-api-access-jkcf2\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.985776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.982158 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d336c29f-9a58-4372-b615-225c9eb1fb4c-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:20.985776 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:20.982170 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b28t\" (UniqueName: \"kubernetes.io/projected/d336c29f-9a58-4372-b615-225c9eb1fb4c-kube-api-access-9b28t\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:21.034973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.034933 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" containerID="cri-o://10c4a0ac81725ca7a4e38281e7f0230d688fc4e6138bb7dea1f2ef6cf957d2cb" gracePeriod=25 Oct 30 19:32:21.466134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.466088 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:32:21.490825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.490786 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:32:21.548703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.548660 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" event={"ID":"d336c29f-9a58-4372-b615-225c9eb1fb4c","Type":"ContainerDied","Data":"f99ec74479acced12ecefabd9b62c8119b945f33863aa2ad2e08b71942224bb5"} Oct 30 19:32:21.549567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.548731 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99ec74479acced12ecefabd9b62c8119b945f33863aa2ad2e08b71942224bb5" Oct 30 19:32:21.549567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.548813 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66" Oct 30 19:32:21.572524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.568876 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" event={"ID":"7e06ff31-d093-4a83-ac9f-13875a3d6e6c","Type":"ContainerDied","Data":"288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc"} Oct 30 19:32:21.573688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.573473 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288dc7f442154bb615f260972689c80c3228b37aef17c930cd29e334c7fa72fc" Oct 30 19:32:21.574185 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:21.570957 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk" Oct 30 19:32:23.018725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.018688 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp"] Oct 30 19:32:23.019496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019464 6907 cpu_manager.go:414] "containerMap: removing container" podUID="7e06ff31-d093-4a83-ac9f-13875a3d6e6c" containerName="revoke-keys" Oct 30 19:32:23.019496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019481 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e06ff31-d093-4a83-ac9f-13875a3d6e6c" containerName="revoke-keys" Oct 30 19:32:23.019630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019521 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b15e6f73-f603-40e4-83de-a4684d6b87f8" containerName="revoke-keys" Oct 30 19:32:23.019630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019527 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15e6f73-f603-40e4-83de-a4684d6b87f8" containerName="revoke-keys" Oct 30 19:32:23.019630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019538 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d336c29f-9a58-4372-b615-225c9eb1fb4c" containerName="revoke-keys" Oct 30 19:32:23.019630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019543 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d336c29f-9a58-4372-b615-225c9eb1fb4c" containerName="revoke-keys" Oct 30 19:32:23.019741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019669 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15e6f73-f603-40e4-83de-a4684d6b87f8" containerName="revoke-keys" Oct 30 19:32:23.019741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019677 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d336c29f-9a58-4372-b615-225c9eb1fb4c" containerName="revoke-keys" Oct 30 19:32:23.019741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.019690 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e06ff31-d093-4a83-ac9f-13875a3d6e6c" containerName="revoke-keys" Oct 30 19:32:23.023805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.023785 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.028038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.028014 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:23.028398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.028378 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:23.041949 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.041918 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp"] Oct 30 19:32:23.087285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.087250 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.087470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.087386 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.087533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.087515 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2zg\" (UniqueName: \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.186599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.186559 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm"] Oct 30 19:32:23.188787 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.188752 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.188938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.188883 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.189048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.189027 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2zg\" (UniqueName: \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.189479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.189421 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.191436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.191417 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.192220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.192205 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.201829 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.201803 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2zg\" (UniqueName: \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\") pod \"maas-api-revoke-sub-c4e899c5-hnsmp\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.204557 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.204535 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm"] Oct 30 19:32:23.291137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.291041 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpvk4\" (UniqueName: \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.291137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.291101 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.291310 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.291187 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.340395 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.340361 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:23.393286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.393247 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpvk4\" (UniqueName: \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.393510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.393316 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.393510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.393347 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.394062 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.394036 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.398723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.398401 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.404669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.404639 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpvk4\" (UniqueName: \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\") pod \"maas-api-revoke-sub-53c6ea93-vnlnm\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.538210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.538176 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:23.769035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.744742 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp"] Oct 30 19:32:23.954678 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:23.954648 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188afca3_d1d3_4e7a_ae42_d12ebc99527d.slice/crio-fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6 WatchSource:0}: Error finding container fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6: Status 404 returned error can't find the container with id fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6 Oct 30 19:32:23.955975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:23.955940 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm"] Oct 30 19:32:24.755248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.755218 6907 generic.go:356] "Generic (PLEG): container finished" podID="cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" containerID="9eb77fc2c55cb0a2999ba6d10ec2b5a8123a19e2f5627d7a84f94b367ac75616" exitCode=0 Oct 30 19:32:24.755605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.755298 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" event={"ID":"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7","Type":"ContainerDied","Data":"9eb77fc2c55cb0a2999ba6d10ec2b5a8123a19e2f5627d7a84f94b367ac75616"} Oct 30 19:32:24.755605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.755364 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" event={"ID":"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7","Type":"ContainerStarted","Data":"bfee98baefee5f6178d3229c7c305e0f2ba3fb056c7d84f562395b29562e7509"} Oct 30 19:32:24.757084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.757062 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" event={"ID":"188afca3-d1d3-4e7a-ae42-d12ebc99527d","Type":"ContainerStarted","Data":"a4a63b5ef0dd7617b0adc91a311d178ca436eb4425d4111b710c9552d4702824"} Oct 30 19:32:24.757202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.757100 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" event={"ID":"188afca3-d1d3-4e7a-ae42-d12ebc99527d","Type":"ContainerStarted","Data":"fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6"} Oct 30 19:32:24.840532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:24.833098 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" podStartSLOduration=1.833075459 podStartE2EDuration="1.833075459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:24.806876579 +0000 UTC m=+750.783102523" watchObservedRunningTime="2026-10-30 19:32:24.833075459 +0000 UTC m=+750.809301405" Oct 30 19:32:25.768607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:25.768568 6907 generic.go:356] "Generic (PLEG): container finished" podID="188afca3-d1d3-4e7a-ae42-d12ebc99527d" containerID="a4a63b5ef0dd7617b0adc91a311d178ca436eb4425d4111b710c9552d4702824" exitCode=0 Oct 30 19:32:25.769024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:25.768649 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" event={"ID":"188afca3-d1d3-4e7a-ae42-d12ebc99527d","Type":"ContainerDied","Data":"a4a63b5ef0dd7617b0adc91a311d178ca436eb4425d4111b710c9552d4702824"} Oct 30 19:32:26.064455 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.064412 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:26.192970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.192936 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\" (UniqueName: \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\") pod \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " Oct 30 19:32:26.193145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.193107 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\") pod \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " Oct 30 19:32:26.193217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.193201 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\") pod \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\" (UID: \"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7\") " Oct 30 19:32:26.193572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.193543 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca" pod "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" (UID: "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:26.193973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.193956 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:26.195088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.195061 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg" pod "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" (UID: "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7"). InnerVolumeSpecName "kube-api-access-hk2zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:26.195164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.195123 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca" pod "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" (UID: "cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:26.295605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.295495 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2zg\" (UniqueName: \"kubernetes.io/projected/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-kube-api-access-hk2zg\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:26.295605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.295535 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:26.467545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.467462 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" containerID="cri-o://7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086" gracePeriod=25 Oct 30 19:32:26.491908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.491829 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" containerID="cri-o://7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc" gracePeriod=25 Oct 30 19:32:26.779392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.779361 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" Oct 30 19:32:26.779778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.779384 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp" event={"ID":"cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7","Type":"ContainerDied","Data":"bfee98baefee5f6178d3229c7c305e0f2ba3fb056c7d84f562395b29562e7509"} Oct 30 19:32:26.779778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.779481 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfee98baefee5f6178d3229c7c305e0f2ba3fb056c7d84f562395b29562e7509" Oct 30 19:32:26.932573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:26.932543 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:27.014527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.014491 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\") pod \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " Oct 30 19:32:27.014729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.014635 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\" (UniqueName: \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\") pod \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " Oct 30 19:32:27.014785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.014746 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\") pod \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\" (UID: \"188afca3-d1d3-4e7a-ae42-d12ebc99527d\") " Oct 30 19:32:27.014926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.014882 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca" pod "188afca3-d1d3-4e7a-ae42-d12ebc99527d" (UID: "188afca3-d1d3-4e7a-ae42-d12ebc99527d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:27.015518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.015499 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.016797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.016764 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4" pod "188afca3-d1d3-4e7a-ae42-d12ebc99527d" (UID: "188afca3-d1d3-4e7a-ae42-d12ebc99527d"). InnerVolumeSpecName "kube-api-access-xpvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:27.016899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.016839 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca" pod "188afca3-d1d3-4e7a-ae42-d12ebc99527d" (UID: "188afca3-d1d3-4e7a-ae42-d12ebc99527d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:27.116974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.116912 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpvk4\" (UniqueName: \"kubernetes.io/projected/188afca3-d1d3-4e7a-ae42-d12ebc99527d-kube-api-access-xpvk4\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.116974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.116980 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188afca3-d1d3-4e7a-ae42-d12ebc99527d-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:27.789029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.788995 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" Oct 30 19:32:27.789538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.788995 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm" event={"ID":"188afca3-d1d3-4e7a-ae42-d12ebc99527d","Type":"ContainerDied","Data":"fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6"} Oct 30 19:32:27.789538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.789092 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda567468942c124a331d5e91bda11c7336218f43e52f10d0f93114797cfdce6" Oct 30 19:32:27.938631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.938594 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:27.938837 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:27.938595 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:28.453164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.453127 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:32:28.453914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.453898 6907 cpu_manager.go:414] "containerMap: removing container" podUID="188afca3-d1d3-4e7a-ae42-d12ebc99527d" containerName="revoke-keys" Oct 30 19:32:28.453979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.453916 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188afca3-d1d3-4e7a-ae42-d12ebc99527d" containerName="revoke-keys" Oct 30 19:32:28.453979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.453963 6907 cpu_manager.go:414] "containerMap: removing container" podUID="cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" containerName="revoke-keys" Oct 30 19:32:28.453979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.453969 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" containerName="revoke-keys" Oct 30 19:32:28.454096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.454086 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" containerName="revoke-keys" Oct 30 19:32:28.454131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.454102 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="188afca3-d1d3-4e7a-ae42-d12ebc99527d" containerName="revoke-keys" Oct 30 19:32:28.458699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.458679 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.472265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.472235 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:32:28.544642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.544605 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcdv\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.544642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.544643 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.544898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.544818 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.544987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.544963 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.545092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.545001 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.545139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.545115 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.545176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.545150 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.545309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.545290 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.545422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.545407 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.551848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.551824 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:32:28.647170 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.647136 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcdv\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.647170 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.647176 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.647412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.647247 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.647412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.647312 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.647412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.647332 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.648235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.648186 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.656127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.656099 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.656423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.656401 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.656629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.656595 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.656880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.656851 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.657080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.657064 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.659030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.659005 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.660918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.660599 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcdv\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.662038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.662010 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.663433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.663394 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.665993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.665954 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.667268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.667231 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.676665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.676642 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\") pod \"e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:28.778000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:28.777966 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:29.073564 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:29.073525 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e697cc8_ad70_4d97_b1da_905022976883.slice/crio-43479c5f10aa2963244ddd03fe0e27f55efae7ea9e3b5a0a135664dd3832ae41 WatchSource:0}: Error finding container 43479c5f10aa2963244ddd03fe0e27f55efae7ea9e3b5a0a135664dd3832ae41: Status 404 returned error can't find the container with id 43479c5f10aa2963244ddd03fe0e27f55efae7ea9e3b5a0a135664dd3832ae41 Oct 30 19:32:29.074332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.074295 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:32:29.076368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.076333 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:29.076478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.076399 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:29.076478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.076432 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:32:29.811927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.811893 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" event={"ID":"8e697cc8-ad70-4d97-b1da-905022976883","Type":"ContainerStarted","Data":"b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015"} Oct 30 19:32:29.812110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.811944 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" event={"ID":"8e697cc8-ad70-4d97-b1da-905022976883","Type":"ContainerStarted","Data":"43479c5f10aa2963244ddd03fe0e27f55efae7ea9e3b5a0a135664dd3832ae41"} Oct 30 19:32:29.839903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:29.839834 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podStartSLOduration=1.8398199929999999 podStartE2EDuration="1.839819993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:29.836361341 +0000 UTC m=+755.812587286" watchObservedRunningTime="2026-10-30 19:32:29.839819993 +0000 UTC m=+755.816045942" Oct 30 19:32:30.778411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:30.778373 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:30.780504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:30.780478 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:30.780566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:30.780529 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:30.987250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:30.987205 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:31.561175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.560964 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" containerID="cri-o://9d0526b62d6c7547c1144fca37f0d56f74d2099c2d99702a9c11201e4aa0f761" gracePeriod=45 Oct 30 19:32:31.710367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.710327 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" containerID="cri-o://cc3549ccc5180a219611eac33806a38d3cf48f93108ee50c10dfea7d2a74acbd" gracePeriod=45 Oct 30 19:32:31.780112 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.779381 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:31.780112 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.779468 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:31.867321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.867285 6907 generic.go:356] "Generic (PLEG): container finished" podID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerID="cc3549ccc5180a219611eac33806a38d3cf48f93108ee50c10dfea7d2a74acbd" exitCode=0 Oct 30 19:32:31.867433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.867341 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" event={"ID":"11df2f79-da70-47f0-b25e-0de02c207bc8","Type":"ContainerDied","Data":"cc3549ccc5180a219611eac33806a38d3cf48f93108ee50c10dfea7d2a74acbd"} Oct 30 19:32:31.869892 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.869864 6907 generic.go:356] "Generic (PLEG): container finished" podID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerID="9d0526b62d6c7547c1144fca37f0d56f74d2099c2d99702a9c11201e4aa0f761" exitCode=0 Oct 30 19:32:31.870632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.870609 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" event={"ID":"92e49551-f63b-4de9-80b1-fd7f4b7283b7","Type":"ContainerDied","Data":"9d0526b62d6c7547c1144fca37f0d56f74d2099c2d99702a9c11201e4aa0f761"} Oct 30 19:32:31.912262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.912233 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.973893 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.973954 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974204 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974259 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974292 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\" (UniqueName: \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974435 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\") pod \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\" (UID: \"92e49551-f63b-4de9-80b1-fd7f4b7283b7\") " Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974631 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974937 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.975395 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.975418 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.976462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.974186 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:31.977132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.976868 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:31.977817 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.977513 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "kube-api-access-l9kvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:31.978687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:31.978603 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs" pod "92e49551-f63b-4de9-80b1-fd7f4b7283b7" (UID: "92e49551-f63b-4de9-80b1-fd7f4b7283b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:32.011844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.011773 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" containerID="cri-o://7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a" gracePeriod=45 Oct 30 19:32:32.077889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.077859 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.078116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.078075 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9kvh\" (UniqueName: \"kubernetes.io/projected/92e49551-f63b-4de9-80b1-fd7f4b7283b7-kube-api-access-l9kvh\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.078116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.078092 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.078116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.078101 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e49551-f63b-4de9-80b1-fd7f4b7283b7-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.151781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.151747 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:32:32.300403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.299840 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.300403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.300018 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.300403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.300167 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.300403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.300317 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.300731 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.300490 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\" (UniqueName: \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.301071 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.300911 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.301239 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.301184 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.301311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.301277 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.301375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.301337 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\") pod \"11df2f79-da70-47f0-b25e-0de02c207bc8\" (UID: \"11df2f79-da70-47f0-b25e-0de02c207bc8\") " Oct 30 19:32:32.302289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.302267 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.302393 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.302293 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.302393 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.302324 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.303534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.303500 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "kube-api-access-s6lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:32.304237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.304214 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:32.305546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.305524 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm" pod "11df2f79-da70-47f0-b25e-0de02c207bc8" (UID: "11df2f79-da70-47f0-b25e-0de02c207bc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.405809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.405719 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6lhf\" (UniqueName: \"kubernetes.io/projected/11df2f79-da70-47f0-b25e-0de02c207bc8-kube-api-access-s6lhf\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.405809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.405755 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11df2f79-da70-47f0-b25e-0de02c207bc8-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.405809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.405772 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11df2f79-da70-47f0-b25e-0de02c207bc8-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.426368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.426338 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:32:32.498555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.498509 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh"] Oct 30 19:32:32.499746 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499723 6907 cpu_manager.go:414] "containerMap: removing container" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" Oct 30 19:32:32.499834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499750 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" Oct 30 19:32:32.499834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499791 6907 cpu_manager.go:414] "containerMap: removing container" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" Oct 30 19:32:32.499834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499798 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" Oct 30 19:32:32.499834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499829 6907 cpu_manager.go:414] "containerMap: removing container" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" Oct 30 19:32:32.500074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.499837 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" Oct 30 19:32:32.500074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.500030 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerName="main" Oct 30 19:32:32.500074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.500052 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" containerName="main" Oct 30 19:32:32.500074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.500071 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" containerName="main" Oct 30 19:32:32.507371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.507336 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.507597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.507576 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.507970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.507947 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.508392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.508006 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.508392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.508185 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.508662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.508348 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.509291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.508383 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.510866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.510844 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.511065 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.511044 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:32.511784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.511747 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\" (UniqueName: \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.511914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.511895 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\") pod \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\" (UID: \"24f80684-9767-4cd0-a7c9-c8058ab5b7fb\") " Oct 30 19:32:32.512754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.512738 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.512840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.512832 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.512905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.512898 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.512954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.512947 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.514943 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.514927 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:32.516836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.516818 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9" pod "24f80684-9767-4cd0-a7c9-c8058ab5b7fb" (UID: "24f80684-9767-4cd0-a7c9-c8058ab5b7fb"). InnerVolumeSpecName "kube-api-access-ck6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:32.518915 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.518900 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:32.519052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.519043 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:32.523372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.523346 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh"] Oct 30 19:32:32.634656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.623325 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.634656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.623582 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68h8\" (UniqueName: \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.634656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.623760 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.634656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.623915 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6m9\" (UniqueName: \"kubernetes.io/projected/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-kube-api-access-ck6m9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.634656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.623935 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f80684-9767-4cd0-a7c9-c8058ab5b7fb-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:32.767481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.764461 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.767481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.764781 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68h8\" (UniqueName: \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.767481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.765384 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.767481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.766363 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.774479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.772118 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.787474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.786436 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:32.787474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.786509 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:32.790481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.789626 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68h8\" (UniqueName: \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.857115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.857073 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:32.880110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.880079 6907 generic.go:356] "Generic (PLEG): container finished" podID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" containerID="7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a" exitCode=0 Oct 30 19:32:32.880238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.880139 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" Oct 30 19:32:32.880238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.880165 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" event={"ID":"24f80684-9767-4cd0-a7c9-c8058ab5b7fb","Type":"ContainerDied","Data":"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a"} Oct 30 19:32:32.880238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.880212 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv" event={"ID":"24f80684-9767-4cd0-a7c9-c8058ab5b7fb","Type":"ContainerDied","Data":"6213b68b8593cafeb61f4384a32985eb9c50e6394c2f242dc78839e3732453fe"} Oct 30 19:32:32.880238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.880236 6907 scope.go:122] "RemoveContainer" containerID="7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a" Oct 30 19:32:32.886795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.886768 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" event={"ID":"92e49551-f63b-4de9-80b1-fd7f4b7283b7","Type":"ContainerDied","Data":"9c6ac4ffcc722cc91faba4efb5ffc7aeff1496929277bf58c53bdba53a83b130"} Oct 30 19:32:32.886907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.886775 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7" Oct 30 19:32:32.888972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.888950 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" event={"ID":"11df2f79-da70-47f0-b25e-0de02c207bc8","Type":"ContainerDied","Data":"4319909cc33af71826b4dbce88844801f43e05b8865e6c322f018665f4fbf049"} Oct 30 19:32:32.889072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.888988 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh" Oct 30 19:32:32.906956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.906932 6907 scope.go:122] "RemoveContainer" containerID="7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a" Oct 30 19:32:32.907298 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:32.907278 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a\": container with ID starting with 7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a not found: ID does not exist" containerID="7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a" Oct 30 19:32:32.907337 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.907308 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a"} err="failed to get container status \"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a\": rpc error: code = NotFound desc = could not find container \"7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a\": container with ID starting with 7c7d2400c0419fb8fa66bce33ea9916c8fdcb8b8357505fb609086f2c0e2d31a not found: ID does not exist" Oct 30 19:32:32.907337 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.907327 6907 scope.go:122] "RemoveContainer" containerID="9d0526b62d6c7547c1144fca37f0d56f74d2099c2d99702a9c11201e4aa0f761" Oct 30 19:32:32.919386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.919174 6907 scope.go:122] "RemoveContainer" containerID="cc3549ccc5180a219611eac33806a38d3cf48f93108ee50c10dfea7d2a74acbd" Oct 30 19:32:32.967406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.967336 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:32:32.973920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.973895 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/premium-w2-d55178-kserve-6b77675f98-pkrb7"] Oct 30 19:32:32.987778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.987744 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:32:32.992842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:32.992819 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-2-w2-d55178-kserve-64f478dc54-dgzjh"] Oct 30 19:32:33.009985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.009957 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:32:33.023595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.023526 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d55178/distinct-w2-d55178-kserve-fc4c9bfc-q4vpv"] Oct 30 19:32:33.187050 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:33.187022 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222ccb1b_fc39_43a9_8943_b9d96d794512.slice/crio-63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94 WatchSource:0}: Error finding container 63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94: Status 404 returned error can't find the container with id 63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94 Oct 30 19:32:33.188240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.188214 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh"] Oct 30 19:32:33.362543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.360624 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp"] Oct 30 19:32:33.400568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.391031 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp"] Oct 30 19:32:33.400568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.391170 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.483190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.483155 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.483393 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.483369 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.483528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.483508 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs982\" (UniqueName: \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.584986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.584947 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.585184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.585167 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.585289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.585271 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs982\" (UniqueName: \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.585970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.585930 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.587983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.587963 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.595899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.595877 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs982\" (UniqueName: \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\") pod \"maas-api-revoke-sub-e302d0d0-jjctp\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.717691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.717605 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:33.780692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.780662 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:33.780845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.780716 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:33.907265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.907230 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" event={"ID":"222ccb1b-fc39-43a9-8943-b9d96d794512","Type":"ContainerStarted","Data":"bd78bfa57a14f8c3aefc26517cb45d529c70478da6f268222708927ea1857547"} Oct 30 19:32:33.907731 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.907293 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" event={"ID":"222ccb1b-fc39-43a9-8943-b9d96d794512","Type":"ContainerStarted","Data":"63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94"} Oct 30 19:32:33.960608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:33.960543 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" podStartSLOduration=1.960522165 podStartE2EDuration="1.960522165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:33.953842761 +0000 UTC m=+759.930068686" watchObservedRunningTime="2026-10-30 19:32:33.960522165 +0000 UTC m=+759.936748110" Oct 30 19:32:34.061620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.061588 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp"] Oct 30 19:32:34.063066 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:34.063033 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5475738_1b37_4a2b_9f3a_0ee4f2eae500.slice/crio-a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6 WatchSource:0}: Error finding container a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6: Status 404 returned error can't find the container with id a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6 Oct 30 19:32:34.162253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.162221 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11df2f79-da70-47f0-b25e-0de02c207bc8" path="/var/lib/kubelet/pods/11df2f79-da70-47f0-b25e-0de02c207bc8/volumes" Oct 30 19:32:34.162792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.162777 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f80684-9767-4cd0-a7c9-c8058ab5b7fb" path="/var/lib/kubelet/pods/24f80684-9767-4cd0-a7c9-c8058ab5b7fb/volumes" Oct 30 19:32:34.163211 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.163199 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e49551-f63b-4de9-80b1-fd7f4b7283b7" path="/var/lib/kubelet/pods/92e49551-f63b-4de9-80b1-fd7f4b7283b7/volumes" Oct 30 19:32:34.778512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.778470 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:34.778704 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.778535 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:34.922292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.922184 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" event={"ID":"d5475738-1b37-4a2b-9f3a-0ee4f2eae500","Type":"ContainerStarted","Data":"b356a955fff2015edb7b341ac1770cb69c3d35693498c59b9042f0e45ccaef94"} Oct 30 19:32:34.922292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.922285 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" event={"ID":"d5475738-1b37-4a2b-9f3a-0ee4f2eae500","Type":"ContainerStarted","Data":"a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6"} Oct 30 19:32:34.923993 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.923965 6907 generic.go:356] "Generic (PLEG): container finished" podID="222ccb1b-fc39-43a9-8943-b9d96d794512" containerID="bd78bfa57a14f8c3aefc26517cb45d529c70478da6f268222708927ea1857547" exitCode=0 Oct 30 19:32:34.924139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.924042 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" event={"ID":"222ccb1b-fc39-43a9-8943-b9d96d794512","Type":"ContainerDied","Data":"bd78bfa57a14f8c3aefc26517cb45d529c70478da6f268222708927ea1857547"} Oct 30 19:32:34.952546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:34.948640 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" podStartSLOduration=1.948621296 podStartE2EDuration="1.948621296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:34.943727722 +0000 UTC m=+760.919953667" watchObservedRunningTime="2026-10-30 19:32:34.948621296 +0000 UTC m=+760.924847244" Oct 30 19:32:35.664253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.664164 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw"] Oct 30 19:32:35.670036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.670004 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-d55178-4ttdw"] Oct 30 19:32:35.779539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.779500 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:35.779831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.779803 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:35.933974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.933874 6907 generic.go:356] "Generic (PLEG): container finished" podID="d5475738-1b37-4a2b-9f3a-0ee4f2eae500" containerID="b356a955fff2015edb7b341ac1770cb69c3d35693498c59b9042f0e45ccaef94" exitCode=0 Oct 30 19:32:35.933974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:35.933960 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" event={"ID":"d5475738-1b37-4a2b-9f3a-0ee4f2eae500","Type":"ContainerDied","Data":"b356a955fff2015edb7b341ac1770cb69c3d35693498c59b9042f0e45ccaef94"} Oct 30 19:32:36.117096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.117067 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:36.154184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.154153 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15e6f73-f603-40e4-83de-a4684d6b87f8" path="/var/lib/kubelet/pods/b15e6f73-f603-40e4-83de-a4684d6b87f8/volumes" Oct 30 19:32:36.207221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.207132 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\") pod \"222ccb1b-fc39-43a9-8943-b9d96d794512\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " Oct 30 19:32:36.207383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.207282 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\") pod \"222ccb1b-fc39-43a9-8943-b9d96d794512\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " Oct 30 19:32:36.207495 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.207387 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\" (UniqueName: \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\") pod \"222ccb1b-fc39-43a9-8943-b9d96d794512\" (UID: \"222ccb1b-fc39-43a9-8943-b9d96d794512\") " Oct 30 19:32:36.207579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.207534 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca" pod "222ccb1b-fc39-43a9-8943-b9d96d794512" (UID: "222ccb1b-fc39-43a9-8943-b9d96d794512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:36.208786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.208758 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:36.209851 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.209828 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca" pod "222ccb1b-fc39-43a9-8943-b9d96d794512" (UID: "222ccb1b-fc39-43a9-8943-b9d96d794512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:36.209959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.209939 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8" pod "222ccb1b-fc39-43a9-8943-b9d96d794512" (UID: "222ccb1b-fc39-43a9-8943-b9d96d794512"). InnerVolumeSpecName "kube-api-access-h68h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:36.309988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.309954 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222ccb1b-fc39-43a9-8943-b9d96d794512-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:36.309988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.309981 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68h8\" (UniqueName: \"kubernetes.io/projected/222ccb1b-fc39-43a9-8943-b9d96d794512-kube-api-access-h68h8\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:36.782466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.782000 6907 patch_prober.go:29] interesting pod/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 19:32:36.782466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.782069 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 19:32:36.947638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.947604 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" Oct 30 19:32:36.948126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.947640 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh" event={"ID":"222ccb1b-fc39-43a9-8943-b9d96d794512","Type":"ContainerDied","Data":"63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94"} Oct 30 19:32:36.948126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:36.947690 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e5b08d4e53b626f4ce1fda04840624cd497ffc245b830d0ef22fe4748adb94" Oct 30 19:32:37.227076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.227046 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:37.284063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.282658 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:32:37.352953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.343331 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\" (UniqueName: \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\") pod \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " Oct 30 19:32:37.352953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.343407 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\") pod \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " Oct 30 19:32:37.352953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.343630 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\") pod \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\" (UID: \"d5475738-1b37-4a2b-9f3a-0ee4f2eae500\") " Oct 30 19:32:37.352953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.351557 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca" pod "d5475738-1b37-4a2b-9f3a-0ee4f2eae500" (UID: "d5475738-1b37-4a2b-9f3a-0ee4f2eae500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:37.387610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.386547 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982" pod "d5475738-1b37-4a2b-9f3a-0ee4f2eae500" (UID: "d5475738-1b37-4a2b-9f3a-0ee4f2eae500"). InnerVolumeSpecName "kube-api-access-qs982". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:37.394801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.394761 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca" pod "d5475738-1b37-4a2b-9f3a-0ee4f2eae500" (UID: "d5475738-1b37-4a2b-9f3a-0ee4f2eae500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:37.452343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.452308 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:37.452343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.452340 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs982\" (UniqueName: \"kubernetes.io/projected/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-kube-api-access-qs982\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:37.452602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.452352 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5475738-1b37-4a2b-9f3a-0ee4f2eae500-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:37.742889 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:37.742850 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:32:37.784620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.784589 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:37.785005 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.784981 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:37.785748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.785726 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:32:37.988469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:37.986410 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:38.060518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.057284 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:38.330525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.325977 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" Oct 30 19:32:38.336463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.331606 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp" event={"ID":"d5475738-1b37-4a2b-9f3a-0ee4f2eae500","Type":"ContainerDied","Data":"a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6"} Oct 30 19:32:38.336463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.331674 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59fd13fe9d55a3a7610866ef044c5e980dbc3c0e54b817db9bce8488bf6e4f6" Oct 30 19:32:38.796572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.792865 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:32:38.796572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:38.793100 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" podUID="d6c99e4d-3238-438b-9288-83580c2f79ea" containerName="maas-api" containerID="cri-o://31facd49f39b853fdd2eaa9c9080e4f6dd214984c34c2cc8aa3b1c4e419400b5" gracePeriod=30 Oct 30 19:32:39.574756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:39.569482 6907 generic.go:356] "Generic (PLEG): container finished" podID="d6c99e4d-3238-438b-9288-83580c2f79ea" containerID="31facd49f39b853fdd2eaa9c9080e4f6dd214984c34c2cc8aa3b1c4e419400b5" exitCode=0 Oct 30 19:32:39.574756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:39.570788 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" event={"ID":"d6c99e4d-3238-438b-9288-83580c2f79ea","Type":"ContainerDied","Data":"31facd49f39b853fdd2eaa9c9080e4f6dd214984c34c2cc8aa3b1c4e419400b5"} Oct 30 19:32:39.689471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:39.651820 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:39.720501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:39.689765 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:32:39.968489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:39.968377 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:40.290783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.290711 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96"] Oct 30 19:32:40.313468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.313399 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-9f2d57-p2c96"] Oct 30 19:32:40.763534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.743348 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:40.765437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.765391 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:40.806686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.778537 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:40.828914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.803594 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:32:40.828914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:40.826683 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:41.084838 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:41.084809 6907 logging.go:55] [core] [Channel #958 SubChannel #959]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.54:50052", ServerName: "10.134.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.54:50052: connect: connection refused" Oct 30 19:32:41.806466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:41.806392 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:41.812606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:41.812549 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:41.816625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:41.816588 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:41.844194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:41.844103 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:32:41.850469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:41.846512 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:42.214584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.138566 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.54:50052\" within 1s: context deadline exceeded" Oct 30 19:32:42.214584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.203727 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:42.214584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.203784 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:42.488824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.366543 6907 patch_prober.go:29] interesting pod/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:42.488824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.366613 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:42.537470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.533715 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f75f0f04-9d22-4efc-98a3-f96ebdc41146" path="/var/lib/kubelet/pods/f75f0f04-9d22-4efc-98a3-f96ebdc41146/volumes" Oct 30 19:32:42.977522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.880655 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:42.977522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:42.883732 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:43.029469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.022955 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 19:32:43.180523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.167103 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:43.180523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.167188 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:43.219585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.184714 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:32:43.219585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.211646 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:43.219585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.211725 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:43.436563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.406744 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:43.436563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.421644 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:43.666897 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.647860 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\") pod \"d6c99e4d-3238-438b-9288-83580c2f79ea\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " Oct 30 19:32:43.819175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.724411 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" containerID="cri-o://e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997" gracePeriod=45 Oct 30 19:32:43.820192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.736298 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca" pod "d6c99e4d-3238-438b-9288-83580c2f79ea" (UID: "d6c99e4d-3238-438b-9288-83580c2f79ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:43.864300 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.864264 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\") pod \"d6c99e4d-3238-438b-9288-83580c2f79ea\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " Oct 30 19:32:43.864628 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.864603 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\" (UniqueName: \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\") pod \"d6c99e4d-3238-438b-9288-83580c2f79ea\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " Oct 30 19:32:43.864835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.864818 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\") pod \"d6c99e4d-3238-438b-9288-83580c2f79ea\" (UID: \"d6c99e4d-3238-438b-9288-83580c2f79ea\") " Oct 30 19:32:43.914561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.895852 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:32:44.049973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.902553 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.943100 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs" pod "d6c99e4d-3238-438b-9288-83580c2f79ea" (UID: "d6c99e4d-3238-438b-9288-83580c2f79ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.002638 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8" pod "d6c99e4d-3238-438b-9288-83580c2f79ea" (UID: "d6c99e4d-3238-438b-9288-83580c2f79ea"). InnerVolumeSpecName "kube-api-access-lksl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.049587 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.051612 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.032224 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" event={"ID":"d56c69d2-5016-4ff6-89c0-1e6962c2a17f","Type":"ContainerDied","Data":"10c4a0ac81725ca7a4e38281e7f0230d688fc4e6138bb7dea1f2ef6cf957d2cb"} Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:43.997972 6907 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6c99e4d-3238-438b-9288-83580c2f79ea-openshift-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.023989 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:44.090866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.032177 6907 generic.go:356] "Generic (PLEG): container finished" podID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerID="10c4a0ac81725ca7a4e38281e7f0230d688fc4e6138bb7dea1f2ef6cf957d2cb" exitCode=0 Oct 30 19:32:44.111549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.104825 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:32:44.111549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.104919 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:44.220481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.220428 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" Oct 30 19:32:44.424571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.406678 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls" pod "d6c99e4d-3238-438b-9288-83580c2f79ea" (UID: "d6c99e4d-3238-438b-9288-83580c2f79ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:44.424571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.414549 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:44.524473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.514724 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": read tcp 10.134.0.2:47778->10.134.0.56:8000: read: connection reset by peer" Oct 30 19:32:44.696536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.671802 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:32:44.770498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.768614 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lksl8\" (UniqueName: \"kubernetes.io/projected/d6c99e4d-3238-438b-9288-83580c2f79ea-kube-api-access-lksl8\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:44.770498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.768673 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-maas-api-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:44.770498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:44.768690 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6c99e4d-3238-438b-9288-83580c2f79ea-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:45.056426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.056080 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:45.056426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.056127 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf" event={"ID":"d6c99e4d-3238-438b-9288-83580c2f79ea","Type":"ContainerDied","Data":"5d14ef2a4890cc927240461beb0184003c67dffbcf38eb82af434b8a249af556"} Oct 30 19:32:45.056426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.056186 6907 scope.go:122] "RemoveContainer" containerID="31facd49f39b853fdd2eaa9c9080e4f6dd214984c34c2cc8aa3b1c4e419400b5" Oct 30 19:32:45.398471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.394655 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:45.398471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.397463 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:32:45.398471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.397698 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" containerID="cri-o://5ce2193af368ac329e6f5da48be3459969f51da391e5380837bffdf6b6df62c2" gracePeriod=30 Oct 30 19:32:45.437610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.437561 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:45.494842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.474870 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:32:45.494842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.475155 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" containerID="cri-o://8a29299f2d26051e3f02da3826ce2de8d43aace88b3023c12433c098a9fa1f08" gracePeriod=30 Oct 30 19:32:45.676895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.676858 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:32:45.709486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.698705 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:32:45.711157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.711109 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" event={"ID":"d56c69d2-5016-4ff6-89c0-1e6962c2a17f","Type":"ContainerDied","Data":"0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9"} Oct 30 19:32:45.782675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.782088 6907 scope.go:122] "RemoveContainer" containerID="10c4a0ac81725ca7a4e38281e7f0230d688fc4e6138bb7dea1f2ef6cf957d2cb" Oct 30 19:32:45.813471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.783354 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\" (UniqueName: \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\") pod \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " Oct 30 19:32:45.912840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.912801 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:32:45.926485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.916734 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" containerID="cri-o://50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432" gracePeriod=30 Oct 30 19:32:45.948135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.948099 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\") pod \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\" (UID: \"d56c69d2-5016-4ff6-89c0-1e6962c2a17f\") " Oct 30 19:32:45.963011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:45.962966 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume" pod "d56c69d2-5016-4ff6-89c0-1e6962c2a17f" (UID: "d56c69d2-5016-4ff6-89c0-1e6962c2a17f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:46.018554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.018514 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:46.109528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.098557 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:46.409469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.399433 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg" pod "d56c69d2-5016-4ff6-89c0-1e6962c2a17f" (UID: "d56c69d2-5016-4ff6-89c0-1e6962c2a17f"). InnerVolumeSpecName "kube-api-access-7sjbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:46.409469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.399570 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:46.648717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.488854 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:46.648717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.623030 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjbg\" (UniqueName: \"kubernetes.io/projected/d56c69d2-5016-4ff6-89c0-1e6962c2a17f-kube-api-access-7sjbg\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:46.655362 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:46.655119 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:32:47.085519 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:47.077572 6907 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613fd7aa_051f_4807_9e5b_9d27a952d84d.slice/crio-conmon-7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice/crio-0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db6bcb9_ee22_4636_a84c_64fbf9e9e1e0.slice/crio-conmon-7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6b2591_c659_41ed_9afe_4509c9232b51.slice/crio-e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613fd7aa_051f_4807_9e5b_9d27a952d84d.slice/crio-7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:32:47.175494 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:47.086961 6907 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6b2591_c659_41ed_9afe_4509c9232b51.slice/crio-conmon-e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice/crio-0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9\": RecentStats: unable to find data in memory cache]" Oct 30 19:32:47.175494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:47.102536 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:32:47.175494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:47.107989 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:47.175494 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:47.126580 6907 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db6bcb9_ee22_4636_a84c_64fbf9e9e1e0.slice/crio-7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6b2591_c659_41ed_9afe_4509c9232b51.slice/crio-conmon-e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice/crio-0aa5c231ffebed50d4c11bf371ad341a9598bd4ec13256dab32040241c5df9f9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee6b2591_c659_41ed_9afe_4509c9232b51.slice/crio-e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db6bcb9_ee22_4636_a84c_64fbf9e9e1e0.slice/crio-conmon-7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c69d2_5016_4ff6_89c0_1e6962c2a17f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:32:47.825251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:47.825204 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:32:47.910540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:47.899190 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": context deadline exceeded" Oct 30 19:32:48.021669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:47.960883 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t" Oct 30 19:32:48.106660 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.106552 6907 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Oct 30 19:32:48.138863 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:48.138835 6907 logging.go:55] [core] [Channel #975 SubChannel #976]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.49:50052", ServerName: "10.134.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.49:50052: connect: connection refused" Oct 30 19:32:48.238985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.227864 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:32:48.238985 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.228006 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 19:32:48.480547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.424784 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:48.480547 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:48.434668 6907 logging.go:55] [core] [Channel #978 SubChannel #979]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: connect: connection refused" Oct 30 19:32:48.659551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.659487 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Oct 30 19:32:48.978032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:48.977475 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.49:50052\" within 1s: context deadline exceeded" Oct 30 19:32:49.072995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.031768 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:49.214633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.214580 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f2d57-c4999cdc-9lzvf"] Oct 30 19:32:49.215619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.215595 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:32:49.230987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.230422 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:49.319176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.318879 6907 generic.go:356] "Generic (PLEG): container finished" podID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerID="7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086" exitCode=0 Oct 30 19:32:49.319176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.319033 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" event={"ID":"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0","Type":"ContainerDied","Data":"7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086"} Oct 30 19:32:49.447712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.433599 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 19:32:49.566568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.542397 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.50:50052\" within 1s: context deadline exceeded" Oct 30 19:32:49.566568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.542580 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:32:49.566568 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:49.543088 6907 logging.go:55] [core] [Channel #978 SubChannel #979]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.50:50052", ServerName: "10.134.0.50:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.50:50052: operation was canceled" Oct 30 19:32:49.566568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.543609 6907 generic.go:356] "Generic (PLEG): container finished" podID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerID="7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc" exitCode=0 Oct 30 19:32:49.566568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.543763 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" event={"ID":"613fd7aa-051f-4807-9e5b-9d27a952d84d","Type":"ContainerDied","Data":"7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc"} Oct 30 19:32:49.876946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.876900 6907 generic.go:356] "Generic (PLEG): container finished" podID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerID="e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997" exitCode=0 Oct 30 19:32:49.877201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.877175 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" event={"ID":"ee6b2591-c659-41ed-9afe-4509c9232b51","Type":"ContainerDied","Data":"e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997"} Oct 30 19:32:49.956724 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.956695 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:32:49.960151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.960127 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:49.972465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:49.960664 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:32:50.138075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.137645 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221537 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221674 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221707 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221827 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221928 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\" (UniqueName: \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.224247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.221986 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\") pod \"ee6b2591-c659-41ed-9afe-4509c9232b51\" (UID: \"ee6b2591-c659-41ed-9afe-4509c9232b51\") " Oct 30 19:32:50.235295 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.235167 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:50.235473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.235372 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:50.241626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.239227 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:50.241626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.239750 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "kube-api-access-z7q4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:50.266669 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.266629 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c99e4d-3238-438b-9288-83580c2f79ea" path="/var/lib/kubelet/pods/d6c99e4d-3238-438b-9288-83580c2f79ea/volumes" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.297722 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw"] Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298820 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d5475738-1b37-4a2b-9f3a-0ee4f2eae500" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298841 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5475738-1b37-4a2b-9f3a-0ee4f2eae500" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298892 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d6c99e4d-3238-438b-9288-83580c2f79ea" containerName="maas-api" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298900 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c99e4d-3238-438b-9288-83580c2f79ea" containerName="maas-api" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298919 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298927 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298943 6907 cpu_manager.go:414] "containerMap: removing container" podUID="222ccb1b-fc39-43a9-8943-b9d96d794512" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298951 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222ccb1b-fc39-43a9-8943-b9d96d794512" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298964 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.298971 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299156 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5475738-1b37-4a2b-9f3a-0ee4f2eae500" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299178 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c99e4d-3238-438b-9288-83580c2f79ea" containerName="maas-api" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299187 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" containerName="payload-pre-processing" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299197 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="222ccb1b-fc39-43a9-8943-b9d96d794512" containerName="revoke-keys" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299208 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.299955 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.302871 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs" pod "ee6b2591-c659-41ed-9afe-4509c9232b51" (UID: "ee6b2591-c659-41ed-9afe-4509c9232b51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:50.309571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.305083 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.310682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.310653 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw"] Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.325130 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.325209 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.325303 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.336970 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7q4s\" (UniqueName: \"kubernetes.io/projected/ee6b2591-c659-41ed-9afe-4509c9232b51-kube-api-access-z7q4s\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.337006 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.337034 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee6b2591-c659-41ed-9afe-4509c9232b51-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.337048 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.337061 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.337075 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee6b2591-c659-41ed-9afe-4509c9232b51-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.344220 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:32:50.357238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.349928 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:32:50.474609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.466429 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:32:50.474609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.467739 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.474609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.467887 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.474609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.468022 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6vl\" (UniqueName: \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.541214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.541134 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9f2d57-55f46ff759-scp2t"] Oct 30 19:32:50.557186 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.557150 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:32:50.557414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.557387 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" containerID="cri-o://67d3879e9d7f74ed76825b86e42f695d2bd829a642ec46fe5cbf503adc9f517b" gracePeriod=30 Oct 30 19:32:50.565235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.564471 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:32:50.565653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.565620 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" containerID="cri-o://1794fca089134a49d49f4e820ca6a9d99625e938764d2a1fd346ca5447c0d085" gracePeriod=30 Oct 30 19:32:50.571543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.571043 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6vl\" (UniqueName: \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.571543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.571336 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.571764 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.571738 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.573413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.573389 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.576137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.576105 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:32:50.576885 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.576861 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" containerID="cri-o://c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d" gracePeriod=30 Oct 30 19:32:50.585706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.585162 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.592218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.592182 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:32:50.594361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.594326 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6vl\" (UniqueName: \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\") pod \"maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.649423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.649382 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:32:50.672614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.672534 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\") pod \"613fd7aa-051f-4807-9e5b-9d27a952d84d\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " Oct 30 19:32:50.672852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.672831 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\" (UniqueName: \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\") pod \"613fd7aa-051f-4807-9e5b-9d27a952d84d\" (UID: \"613fd7aa-051f-4807-9e5b-9d27a952d84d\") " Oct 30 19:32:50.672955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.672931 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume" pod "613fd7aa-051f-4807-9e5b-9d27a952d84d" (UID: "613fd7aa-051f-4807-9e5b-9d27a952d84d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:50.673513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.673491 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613fd7aa-051f-4807-9e5b-9d27a952d84d-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.675534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.675498 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv" pod "613fd7aa-051f-4807-9e5b-9d27a952d84d" (UID: "613fd7aa-051f-4807-9e5b-9d27a952d84d"). InnerVolumeSpecName "kube-api-access-mcwpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:50.720405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.720378 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:32:50.775665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.775623 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwpv\" (UniqueName: \"kubernetes.io/projected/613fd7aa-051f-4807-9e5b-9d27a952d84d-kube-api-access-mcwpv\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.796654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.796170 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:50.880807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.879425 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\") pod \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " Oct 30 19:32:50.880807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.879716 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\" (UniqueName: \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\") pod \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\" (UID: \"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0\") " Oct 30 19:32:50.927642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.881534 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume" pod "9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" (UID: "9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:50.927642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.927505 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq" pod "9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" (UID: "9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0"). InnerVolumeSpecName "kube-api-access-4c5vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:50.944066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.944038 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p"] Oct 30 19:32:50.946486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946459 6907 cpu_manager.go:414] "containerMap: removing container" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" Oct 30 19:32:50.946572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946489 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" Oct 30 19:32:50.946572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946533 6907 cpu_manager.go:414] "containerMap: removing container" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" Oct 30 19:32:50.946572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946543 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" Oct 30 19:32:50.946766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946752 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" containerName="payload-pre-processing" Oct 30 19:32:50.946821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.946773 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" containerName="payload-processing" Oct 30 19:32:50.956233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.954963 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" Oct 30 19:32:50.956233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.955115 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:32:50.960809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.956636 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l" event={"ID":"9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0","Type":"ContainerDied","Data":"c83613a86305308be8e457c65b594a257b3ac50aee48a931d6cf382973146245"} Oct 30 19:32:50.960809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.956687 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p"] Oct 30 19:32:50.960809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.956777 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:50.960809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.956782 6907 scope.go:122] "RemoveContainer" containerID="7821dd2db5455f74909f60518feadad62c5b5beee1e84538eb063cc9ac571086" Oct 30 19:32:50.963965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.963889 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" event={"ID":"613fd7aa-051f-4807-9e5b-9d27a952d84d","Type":"ContainerDied","Data":"5ed426a6b0023f45b6fcd3ffc1bd1824ac8e5b7d1601271945e9b625b015602d"} Oct 30 19:32:50.964354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.964173 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27" Oct 30 19:32:50.969768 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.969735 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" event={"ID":"ee6b2591-c659-41ed-9afe-4509c9232b51","Type":"ContainerDied","Data":"6b4821649f3c06a100122061571b3d6ea16a30819ae4f9c9988d1b2d1937ada0"} Oct 30 19:32:50.969872 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.969818 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm" Oct 30 19:32:50.979482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.979339 6907 generic.go:356] "Generic (PLEG): container finished" podID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerID="8a29299f2d26051e3f02da3826ce2de8d43aace88b3023c12433c098a9fa1f08" exitCode=0 Oct 30 19:32:50.980133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.979661 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" event={"ID":"e62fcbd8-e151-4e4d-a525-8f08904e222c","Type":"ContainerDied","Data":"8a29299f2d26051e3f02da3826ce2de8d43aace88b3023c12433c098a9fa1f08"} Oct 30 19:32:50.983015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.982958 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5vq\" (UniqueName: \"kubernetes.io/projected/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-kube-api-access-4c5vq\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.983015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.982984 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:50.991964 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.991945 6907 scope.go:122] "RemoveContainer" containerID="7b912283629102e52e312406ee054c6754bc1a1ca0d6571fa7271bc3b5f594dc" Oct 30 19:32:50.992660 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.992634 6907 generic.go:356] "Generic (PLEG): container finished" podID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerID="5ce2193af368ac329e6f5da48be3459969f51da391e5380837bffdf6b6df62c2" exitCode=0 Oct 30 19:32:50.992756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.992688 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" Oct 30 19:32:50.992846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:50.992706 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8" event={"ID":"342c0ddb-5be4-46be-8ee5-704bad5babf8","Type":"ContainerDied","Data":"5ce2193af368ac329e6f5da48be3459969f51da391e5380837bffdf6b6df62c2"} Oct 30 19:32:51.046413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.046385 6907 scope.go:122] "RemoveContainer" containerID="e5892d348af6b86b756a36bc2b47e18d9d2292e6d168d93e427c1a25b9092997" Oct 30 19:32:51.051682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.051503 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:32:51.051682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.051622 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:32:51.059063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.059026 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d55178-9fd485587-rjd9l"] Oct 30 19:32:51.063506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.063438 6907 scope.go:122] "RemoveContainer" containerID="5ce2193af368ac329e6f5da48be3459969f51da391e5380837bffdf6b6df62c2" Oct 30 19:32:51.084010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.083982 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:32:51.084386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084361 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.084520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084503 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.084569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084555 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.084601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084590 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.084724 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084695 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.084873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084747 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.085028 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084968 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.085028 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.084975 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.085110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085053 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.085146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085133 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.085215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085196 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\") pod \"342c0ddb-5be4-46be-8ee5-704bad5babf8\" (UID: \"342c0ddb-5be4-46be-8ee5-704bad5babf8\") " Oct 30 19:32:51.085373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085350 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:51.085642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085618 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.085812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085792 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.085998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.085881 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4ck\" (UniqueName: \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.086161 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086143 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.086266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086252 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.086411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086396 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.086540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086416 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.086540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086431 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.086540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086459 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342c0ddb-5be4-46be-8ee5-704bad5babf8-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.086540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.086474 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.087168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.087140 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "kube-api-access-tqhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.087249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.087153 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.087249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.087165 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.087436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.087419 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo" pod "342c0ddb-5be4-46be-8ee5-704bad5babf8" (UID: "342c0ddb-5be4-46be-8ee5-704bad5babf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:51.090302 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.090282 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d55178-55c954767f-qpd27"] Oct 30 19:32:51.101253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.101226 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:32:51.113255 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.113214 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-171808/premium-w0-171808-kserve-fbbdbdd9c-7xnjm"] Oct 30 19:32:51.166929 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:51.166892 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd187816_4859_45ca_a89b_831c75b4eed0.slice/crio-17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9 WatchSource:0}: Error finding container 17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9: Status 404 returned error can't find the container with id 17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9 Oct 30 19:32:51.168174 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.168151 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw"] Oct 30 19:32:51.187809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.187782 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.187927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.187841 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.187927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.187900 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188020 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188073 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188141 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.188300 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188274 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.188368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188286 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188354 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188387 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188553 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\") pod \"e62fcbd8-e151-4e4d-a525-8f08904e222c\" (UID: \"e62fcbd8-e151-4e4d-a525-8f08904e222c\") " Oct 30 19:32:51.188612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188579 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.188730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188635 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:51.188730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188707 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.188835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188818 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.188987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.188965 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4ck\" (UniqueName: \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.189166 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189149 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.189314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189296 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189319 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhsr\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-kube-api-access-tqhsr\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189334 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189347 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189360 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189380 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189373 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189385 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342c0ddb-5be4-46be-8ee5-704bad5babf8-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189397 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189410 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e62fcbd8-e151-4e4d-a525-8f08904e222c-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.189849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.189635 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.191483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.191353 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:51.191483 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.191359 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.192139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.192118 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "kube-api-access-txmkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.192847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.192821 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.193018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.192958 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy" pod "e62fcbd8-e151-4e4d-a525-8f08904e222c" (UID: "e62fcbd8-e151-4e4d-a525-8f08904e222c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.202218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.202192 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4ck\" (UniqueName: \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\") pod \"maas-api-revoke-sub-f03ff582-l9c9p\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.290608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.290576 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.290608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.290608 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.290608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.290622 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txmkm\" (UniqueName: \"kubernetes.io/projected/e62fcbd8-e151-4e4d-a525-8f08904e222c-kube-api-access-txmkm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.290845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.290635 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e62fcbd8-e151-4e4d-a525-8f08904e222c-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.293832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.293807 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:51.322562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.322536 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:32:51.331360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.331329 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-sq4j8"] Oct 30 19:32:51.331612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.331587 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:32:51.494089 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494047 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494168 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494239 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494272 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494322 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494332 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494435 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494496 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.494551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494525 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.495008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494603 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.495008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494708 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\") pod \"06b7a44c-d384-4f50-a740-fffe46c2ed46\" (UID: \"06b7a44c-d384-4f50-a740-fffe46c2ed46\") " Oct 30 19:32:51.495008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.494955 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.495194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495086 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:51.495194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495178 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.495612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495586 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.495709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495618 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b7a44c-d384-4f50-a740-fffe46c2ed46-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.495709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495634 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.495709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495649 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.495709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.495664 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.496916 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.496896 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:51.496995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.496923 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:51.496995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.496960 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "kube-api-access-xs5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.497058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.497012 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token" pod "06b7a44c-d384-4f50-a740-fffe46c2ed46" (UID: "06b7a44c-d384-4f50-a740-fffe46c2ed46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:51.605480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.598823 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.605480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.598871 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.605480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.598887 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5tk\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-kube-api-access-xs5tk\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.605480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.598900 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b7a44c-d384-4f50-a740-fffe46c2ed46-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:51.633656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:51.633619 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p"] Oct 30 19:32:51.638287 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:32:51.638257 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77dd963_bc60_424e_85fb_a2035f2dcad0.slice/crio-4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76 WatchSource:0}: Error finding container 4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76: Status 404 returned error can't find the container with id 4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76 Oct 30 19:32:52.007920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.007872 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" event={"ID":"e62fcbd8-e151-4e4d-a525-8f08904e222c","Type":"ContainerDied","Data":"9afd367540aac80017a45ff4ed085e9ac9a2506f2a6efc9a5137e560182ce133"} Oct 30 19:32:52.008095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.007948 6907 scope.go:122] "RemoveContainer" containerID="8a29299f2d26051e3f02da3826ce2de8d43aace88b3023c12433c098a9fa1f08" Oct 30 19:32:52.008095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.007895 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7" Oct 30 19:32:52.018613 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.018581 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" event={"ID":"f77dd963-bc60-424e-85fb-a2035f2dcad0","Type":"ContainerStarted","Data":"b5ba594d360b56cbbc769b4179444447d5bd94e16e3f675d83c84d1eafae970b"} Oct 30 19:32:52.018710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.018636 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" event={"ID":"f77dd963-bc60-424e-85fb-a2035f2dcad0","Type":"ContainerStarted","Data":"4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76"} Oct 30 19:32:52.020290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.020263 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" event={"ID":"dd187816-4859-45ca-a89b-831c75b4eed0","Type":"ContainerStarted","Data":"df411df8404e91172ef297afb9eeca3057f9947c75aefb2d6f948c2acba8b1b0"} Oct 30 19:32:52.020388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.020310 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" event={"ID":"dd187816-4859-45ca-a89b-831c75b4eed0","Type":"ContainerStarted","Data":"17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9"} Oct 30 19:32:52.021867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.021840 6907 generic.go:356] "Generic (PLEG): container finished" podID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerID="50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432" exitCode=0 Oct 30 19:32:52.021975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.021873 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" Oct 30 19:32:52.021975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.021914 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" event={"ID":"06b7a44c-d384-4f50-a740-fffe46c2ed46","Type":"ContainerDied","Data":"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432"} Oct 30 19:32:52.021975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.021960 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d" event={"ID":"06b7a44c-d384-4f50-a740-fffe46c2ed46","Type":"ContainerDied","Data":"6d312fd1864e1718b2e6577a7e7408943935c4d119a8eb122a4761c31d87ddfe"} Oct 30 19:32:52.022600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.022580 6907 scope.go:122] "RemoveContainer" containerID="50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432" Oct 30 19:32:52.036741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.036720 6907 scope.go:122] "RemoveContainer" containerID="50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432" Oct 30 19:32:52.037124 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:52.037101 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432\": container with ID starting with 50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432 not found: ID does not exist" containerID="50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432" Oct 30 19:32:52.037241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.037138 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432"} err="failed to get container status \"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432\": rpc error: code = NotFound desc = could not find container \"50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432\": container with ID starting with 50ee571b1c3bf4885a8783d251fe32f22a9df6ddd4bda5aaec89e105648cf432 not found: ID does not exist" Oct 30 19:32:52.038268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.038238 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:32:52.043233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.043210 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-2lcd7"] Oct 30 19:32:52.065273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.065244 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:32:52.069746 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.069722 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d55178-openshift-default-5c669c9fdf-hzz7d"] Oct 30 19:32:52.078926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.078869 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" podStartSLOduration=2.078849924 podStartE2EDuration="2.078849924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:52.073792348 +0000 UTC m=+778.050018294" watchObservedRunningTime="2026-10-30 19:32:52.078849924 +0000 UTC m=+778.055075869" Oct 30 19:32:52.098133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.098030 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" podStartSLOduration=6.098011756 podStartE2EDuration="6.098011756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:52.090101732 +0000 UTC m=+778.066327680" watchObservedRunningTime="2026-10-30 19:32:52.098011756 +0000 UTC m=+778.074237701" Oct 30 19:32:52.153318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.153282 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" path="/var/lib/kubelet/pods/06b7a44c-d384-4f50-a740-fffe46c2ed46/volumes" Oct 30 19:32:52.154166 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.154147 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" path="/var/lib/kubelet/pods/342c0ddb-5be4-46be-8ee5-704bad5babf8/volumes" Oct 30 19:32:52.154648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.154631 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613fd7aa-051f-4807-9e5b-9d27a952d84d" path="/var/lib/kubelet/pods/613fd7aa-051f-4807-9e5b-9d27a952d84d/volumes" Oct 30 19:32:52.155523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.155504 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0" path="/var/lib/kubelet/pods/9db6bcb9-ee22-4636-a84c-64fbf9e9e1e0/volumes" Oct 30 19:32:52.155953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.155937 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56c69d2-5016-4ff6-89c0-1e6962c2a17f" path="/var/lib/kubelet/pods/d56c69d2-5016-4ff6-89c0-1e6962c2a17f/volumes" Oct 30 19:32:52.156359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.156347 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" path="/var/lib/kubelet/pods/e62fcbd8-e151-4e4d-a525-8f08904e222c/volumes" Oct 30 19:32:52.157139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.157124 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6b2591-c659-41ed-9afe-4509c9232b51" path="/var/lib/kubelet/pods/ee6b2591-c659-41ed-9afe-4509c9232b51/volumes" Oct 30 19:32:52.318006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.317964 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" containerID="cri-o://4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7" gracePeriod=45 Oct 30 19:32:52.555431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.555384 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:32:52.555431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.555466 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:32:52.677181 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.677065 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:52.747075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747040 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\" (UniqueName: \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.747269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747111 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.747269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747172 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.747269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747214 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.747269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747257 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.747547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.747432 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\") pod \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\" (UID: \"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7\") " Oct 30 19:32:52.748409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.748362 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:52.748602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.748375 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:52.748766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.748743 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:52.751498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.751090 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:52.751603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.751487 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:52.752923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.752899 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4" pod "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" (UID: "56b73626-6cd2-4d5c-bdea-abc9efe3f2b7"). InnerVolumeSpecName "kube-api-access-fw2t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:52.849678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849638 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:52.849678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849671 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2t4\" (UniqueName: \"kubernetes.io/projected/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-kube-api-access-fw2t4\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:52.849896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849685 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:52.849896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849699 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:52.849896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849712 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:52.849896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:52.849723 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:53.032631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.032536 6907 generic.go:356] "Generic (PLEG): container finished" podID="f77dd963-bc60-424e-85fb-a2035f2dcad0" containerID="b5ba594d360b56cbbc769b4179444447d5bd94e16e3f675d83c84d1eafae970b" exitCode=0 Oct 30 19:32:53.032804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.032632 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" event={"ID":"f77dd963-bc60-424e-85fb-a2035f2dcad0","Type":"ContainerDied","Data":"b5ba594d360b56cbbc769b4179444447d5bd94e16e3f675d83c84d1eafae970b"} Oct 30 19:32:53.034782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.034753 6907 generic.go:356] "Generic (PLEG): container finished" podID="dd187816-4859-45ca-a89b-831c75b4eed0" containerID="df411df8404e91172ef297afb9eeca3057f9947c75aefb2d6f948c2acba8b1b0" exitCode=0 Oct 30 19:32:53.034928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.034839 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" event={"ID":"dd187816-4859-45ca-a89b-831c75b4eed0","Type":"ContainerDied","Data":"df411df8404e91172ef297afb9eeca3057f9947c75aefb2d6f948c2acba8b1b0"} Oct 30 19:32:53.038824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.038799 6907 generic.go:356] "Generic (PLEG): container finished" podID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerID="4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7" exitCode=0 Oct 30 19:32:53.038963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.038846 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" Oct 30 19:32:53.038963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.038864 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" event={"ID":"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7","Type":"ContainerDied","Data":"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7"} Oct 30 19:32:53.038963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.038899 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq" event={"ID":"56b73626-6cd2-4d5c-bdea-abc9efe3f2b7","Type":"ContainerDied","Data":"6d0de7cfcc7371119cabc580a01cf6d4516c3559da51cb8d08e636ceaf72daf3"} Oct 30 19:32:53.038963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.038917 6907 scope.go:122] "RemoveContainer" containerID="4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7" Oct 30 19:32:53.056309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.056282 6907 scope.go:122] "RemoveContainer" containerID="4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7" Oct 30 19:32:53.057019 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:53.056686 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7\": container with ID starting with 4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7 not found: ID does not exist" containerID="4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7" Oct 30 19:32:53.057019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.056733 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7"} err="failed to get container status \"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7\": rpc error: code = NotFound desc = could not find container \"4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7\": container with ID starting with 4737d2a15e40b28976ee579b7cba400d9f32bd2fd89454da93697baffa7459c7 not found: ID does not exist" Oct 30 19:32:53.093771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.093738 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:32:53.098305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.098276 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-639c15/distinct-w6-639c15-kserve-6c8c4647f8-mg7dq"] Oct 30 19:32:53.173168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.173127 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:32:53.241754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.241718 6907 patch_prober.go:29] interesting pod/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:32:53.241925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:53.241773 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:32:54.161176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.159810 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" path="/var/lib/kubelet/pods/56b73626-6cd2-4d5c-bdea-abc9efe3f2b7/volumes" Oct 30 19:32:54.221019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.220987 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:54.274868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.274828 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:54.275053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.274901 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\") pod \"f77dd963-bc60-424e-85fb-a2035f2dcad0\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " Oct 30 19:32:54.275121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.275091 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\") pod \"f77dd963-bc60-424e-85fb-a2035f2dcad0\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " Oct 30 19:32:54.275184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.275169 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\" (UniqueName: \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\") pod \"f77dd963-bc60-424e-85fb-a2035f2dcad0\" (UID: \"f77dd963-bc60-424e-85fb-a2035f2dcad0\") " Oct 30 19:32:54.275345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.275316 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca" pod "f77dd963-bc60-424e-85fb-a2035f2dcad0" (UID: "f77dd963-bc60-424e-85fb-a2035f2dcad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:54.276419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.276380 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.284432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.281589 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca" pod "f77dd963-bc60-424e-85fb-a2035f2dcad0" (UID: "f77dd963-bc60-424e-85fb-a2035f2dcad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:54.286332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.286290 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck" pod "f77dd963-bc60-424e-85fb-a2035f2dcad0" (UID: "f77dd963-bc60-424e-85fb-a2035f2dcad0"). InnerVolumeSpecName "kube-api-access-nb4ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:54.377642 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.377549 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\" (UniqueName: \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\") pod \"dd187816-4859-45ca-a89b-831c75b4eed0\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " Oct 30 19:32:54.377807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.377717 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\") pod \"dd187816-4859-45ca-a89b-831c75b4eed0\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " Oct 30 19:32:54.377882 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.377866 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\") pod \"dd187816-4859-45ca-a89b-831c75b4eed0\" (UID: \"dd187816-4859-45ca-a89b-831c75b4eed0\") " Oct 30 19:32:54.378083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.378057 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca" pod "dd187816-4859-45ca-a89b-831c75b4eed0" (UID: "dd187816-4859-45ca-a89b-831c75b4eed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:54.378341 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.378326 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4ck\" (UniqueName: \"kubernetes.io/projected/f77dd963-bc60-424e-85fb-a2035f2dcad0-kube-api-access-nb4ck\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.378388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.378344 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.378388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.378356 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f77dd963-bc60-424e-85fb-a2035f2dcad0-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.379799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.379775 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl" pod "dd187816-4859-45ca-a89b-831c75b4eed0" (UID: "dd187816-4859-45ca-a89b-831c75b4eed0"). InnerVolumeSpecName "kube-api-access-6c6vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:54.379799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.379786 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca" pod "dd187816-4859-45ca-a89b-831c75b4eed0" (UID: "dd187816-4859-45ca-a89b-831c75b4eed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:54.480482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.480400 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd187816-4859-45ca-a89b-831c75b4eed0-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:54.480482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:54.480436 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6vl\" (UniqueName: \"kubernetes.io/projected/dd187816-4859-45ca-a89b-831c75b4eed0-kube-api-access-6c6vl\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.229105 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" event={"ID":"f77dd963-bc60-424e-85fb-a2035f2dcad0","Type":"ContainerDied","Data":"4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76"} Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.229170 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0b6fdb12bbe09ceb95fa46cd4ca398ec5dba6f8f2296d021cccbfc55f2bf76" Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.229269 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p" Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.236551 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" event={"ID":"dd187816-4859-45ca-a89b-831c75b4eed0","Type":"ContainerDied","Data":"17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9"} Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.236707 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17192ac0bff4fbcbb05955eb0091b522464cb9ceb91826b2f55c39d7f0e426e9" Oct 30 19:32:55.251498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.236892 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw" Oct 30 19:32:55.351107 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:55.350960 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" containerID="cri-o://10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe" gracePeriod=25 Oct 30 19:32:56.242777 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.242454 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:32:56.246063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.246030 6907 generic.go:356] "Generic (PLEG): container finished" podID="81102270-6923-4e17-8da4-4b3829a3249b" containerID="67d3879e9d7f74ed76825b86e42f695d2bd829a642ec46fe5cbf503adc9f517b" exitCode=0 Oct 30 19:32:56.246199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.246155 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" event={"ID":"81102270-6923-4e17-8da4-4b3829a3249b","Type":"ContainerDied","Data":"67d3879e9d7f74ed76825b86e42f695d2bd829a642ec46fe5cbf503adc9f517b"} Oct 30 19:32:56.248269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.248241 6907 generic.go:356] "Generic (PLEG): container finished" podID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerID="c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d" exitCode=0 Oct 30 19:32:56.248529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.248506 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerDied","Data":"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d"} Oct 30 19:32:56.248661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.248518 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" Oct 30 19:32:56.248902 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.248746 6907 scope.go:122] "RemoveContainer" containerID="c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d" Oct 30 19:32:56.248969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.248654 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd" event={"ID":"2e109c52-c2fd-49b8-9172-374d5b43f3ea","Type":"ContainerDied","Data":"3c8f65d36b1600ae5e89e25b3721feb40fdbc76ce98d245ed2ab211eea58ccbb"} Oct 30 19:32:56.252721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.252636 6907 generic.go:356] "Generic (PLEG): container finished" podID="44654834-b2a2-41ca-b840-89809c0fea2e" containerID="1794fca089134a49d49f4e820ca6a9d99625e938764d2a1fd346ca5447c0d085" exitCode=0 Oct 30 19:32:56.252721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.252680 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" event={"ID":"44654834-b2a2-41ca-b840-89809c0fea2e","Type":"ContainerDied","Data":"1794fca089134a49d49f4e820ca6a9d99625e938764d2a1fd346ca5447c0d085"} Oct 30 19:32:56.266101 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.266078 6907 scope.go:122] "RemoveContainer" containerID="922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.284332 6907 scope.go:122] "RemoveContainer" containerID="c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285355 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285408 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285461 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285490 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285565 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285726 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285831 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.285863 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.286005 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\") pod \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\" (UID: \"2e109c52-c2fd-49b8-9172-374d5b43f3ea\") " Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.287070 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.287331 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.288396 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.288742 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:56.288865 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d\": container with ID starting with c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d not found: ID does not exist" containerID="c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.288899 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d"} err="failed to get container status \"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d\": rpc error: code = NotFound desc = could not find container \"c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d\": container with ID starting with c2c016caab55d912a0f059a10982966044647718e8e6517d71431fec835a6e4d not found: ID does not exist" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.288926 6907 scope.go:122] "RemoveContainer" containerID="922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:32:56.290063 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f\": container with ID starting with 922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f not found: ID does not exist" containerID="922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.290097 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f"} err="failed to get container status \"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f\": rpc error: code = NotFound desc = could not find container \"922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f\": container with ID starting with 922dd1095ba17874117134e99fa26de3c18328688ebb4a25d838b0ab428b5f1f not found: ID does not exist" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.290138 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.291367 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "kube-api-access-2w22l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.296606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.294295 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.301489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.301454 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:56.309720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.308687 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy" pod "2e109c52-c2fd-49b8-9172-374d5b43f3ea" (UID: "2e109c52-c2fd-49b8-9172-374d5b43f3ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.342278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.342250 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:32:56.367968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.367938 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:32:56.387279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387238 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387321 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387378 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387482 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387602 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387541 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387669 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387684 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.387797 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387753 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.387905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.387874 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.388036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.388011 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.388107 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.388086 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\") pod \"44654834-b2a2-41ca-b840-89809c0fea2e\" (UID: \"44654834-b2a2-41ca-b840-89809c0fea2e\") " Oct 30 19:32:56.388352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.388330 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.388869 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.388816 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:56.389026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389004 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.389473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389435 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.389607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389593 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.389698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389685 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.389784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389772 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.389864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389853 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.389943 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.389931 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w22l\" (UniqueName: \"kubernetes.io/projected/2e109c52-c2fd-49b8-9172-374d5b43f3ea-kube-api-access-2w22l\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.394838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.392830 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "kube-api-access-gbzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.394838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.393024 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:56.394838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.393213 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.398031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398006 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy" pod "44654834-b2a2-41ca-b840-89809c0fea2e" (UID: "44654834-b2a2-41ca-b840-89809c0fea2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.398546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398513 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398547 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44654834-b2a2-41ca-b840-89809c0fea2e-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398563 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398578 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398594 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398610 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398624 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.398665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.398637 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e109c52-c2fd-49b8-9172-374d5b43f3ea-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.499220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.499180 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.499664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.499400 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.499812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.499727 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.499812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.499488 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.500388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500361 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500427 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500503 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500562 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500595 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500662 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\") pod \"81102270-6923-4e17-8da4-4b3829a3249b\" (UID: \"81102270-6923-4e17-8da4-4b3829a3249b\") " Oct 30 19:32:56.500793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.500687 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.501321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501297 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzms\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-kube-api-access-gbzms\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501325 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501341 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44654834-b2a2-41ca-b840-89809c0fea2e-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501355 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44654834-b2a2-41ca-b840-89809c0fea2e-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501368 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501381 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44654834-b2a2-41ca-b840-89809c0fea2e-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.501749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501624 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.501749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501721 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:56.501865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.501402 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.502841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.502814 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.503066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.503042 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "kube-api-access-sccbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:56.503603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.503577 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:32:56.503902 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.503631 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy" pod "81102270-6923-4e17-8da4-4b3829a3249b" (UID: "81102270-6923-4e17-8da4-4b3829a3249b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:56.575733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.575688 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:32:56.580786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.580758 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-78rhd"] Oct 30 19:32:56.603110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603082 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sccbj\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-kube-api-access-sccbj\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603110 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81102270-6923-4e17-8da4-4b3829a3249b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603119 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81102270-6923-4e17-8da4-4b3829a3249b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603131 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603139 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81102270-6923-4e17-8da4-4b3829a3249b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603149 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.603284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.603157 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81102270-6923-4e17-8da4-4b3829a3249b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:56.642973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.635091 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:32:56.642973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:56.642613 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:32:57.268917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.268883 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" Oct 30 19:32:57.268917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.268907 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn" event={"ID":"44654834-b2a2-41ca-b840-89809c0fea2e","Type":"ContainerDied","Data":"1f1c87c16262443b32b5401007839c5670059e115afbd684624f4b30f1ae8e1f"} Oct 30 19:32:57.269435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.268976 6907 scope.go:122] "RemoveContainer" containerID="1794fca089134a49d49f4e820ca6a9d99625e938764d2a1fd346ca5447c0d085" Oct 30 19:32:57.270668 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.270644 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" event={"ID":"81102270-6923-4e17-8da4-4b3829a3249b","Type":"ContainerDied","Data":"b4ace794d67a2dd41a8cdfad690f1e50fc105647017efd5ddd08b99daabf06a7"} Oct 30 19:32:57.270774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.270693 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8" Oct 30 19:32:57.280909 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.280890 6907 scope.go:122] "RemoveContainer" containerID="67d3879e9d7f74ed76825b86e42f695d2bd829a642ec46fe5cbf503adc9f517b" Oct 30 19:32:57.299070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.299042 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:32:57.303631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.303605 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-rm2qn"] Oct 30 19:32:57.314789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.314257 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:32:57.319655 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.319628 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f2d57-openshift-default-79bc845645-9wjl8"] Oct 30 19:32:57.980368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.976855 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:32:57.980368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.977123 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" containerID="cri-o://b7fb7f26f2811b1c7dda37e1b4796090293b42051551224cd3cc4c8f1c99aa49" gracePeriod=30 Oct 30 19:32:57.984552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.984524 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:32:57.984892 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.984757 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" containerID="cri-o://1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010" gracePeriod=30 Oct 30 19:32:57.985150 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.985118 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:32:57.991592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.991565 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:32:57.991834 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.991803 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" containerID="cri-o://663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872" gracePeriod=30 Oct 30 19:32:57.998640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.998611 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:32:57.998868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:57.998844 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" containerID="cri-o://d4f344e11cb38fa5d78f717c752814572286fd1cc4abf901f3f75b3e425f2096" gracePeriod=30 Oct 30 19:32:58.283597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:58.283500 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" path="/var/lib/kubelet/pods/2e109c52-c2fd-49b8-9172-374d5b43f3ea/volumes" Oct 30 19:32:58.284402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:58.284376 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" path="/var/lib/kubelet/pods/44654834-b2a2-41ca-b840-89809c0fea2e/volumes" Oct 30 19:32:58.284989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:58.284973 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81102270-6923-4e17-8da4-4b3829a3249b" path="/var/lib/kubelet/pods/81102270-6923-4e17-8da4-4b3829a3249b/volumes" Oct 30 19:32:58.527132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:58.527091 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:32:59.594610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.594572 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm"] Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595203 6907 cpu_manager.go:414] "containerMap: removing container" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595215 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595229 6907 cpu_manager.go:414] "containerMap: removing container" podUID="dd187816-4859-45ca-a89b-831c75b4eed0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595234 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd187816-4859-45ca-a89b-831c75b4eed0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595246 6907 cpu_manager.go:414] "containerMap: removing container" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595251 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595262 6907 cpu_manager.go:414] "containerMap: removing container" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595267 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595282 6907 cpu_manager.go:414] "containerMap: removing container" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595288 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595297 6907 cpu_manager.go:414] "containerMap: removing container" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595302 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595315 6907 cpu_manager.go:414] "containerMap: removing container" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595319 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595335 6907 cpu_manager.go:414] "containerMap: removing container" podUID="f77dd963-bc60-424e-85fb-a2035f2dcad0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595340 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f77dd963-bc60-424e-85fb-a2035f2dcad0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595347 6907 cpu_manager.go:414] "containerMap: removing container" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595352 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595369 6907 cpu_manager.go:414] "containerMap: removing container" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595373 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595476 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="f77dd963-bc60-424e-85fb-a2035f2dcad0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595486 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="44654834-b2a2-41ca-b840-89809c0fea2e" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595494 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b73626-6cd2-4d5c-bdea-abc9efe3f2b7" containerName="main" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595500 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="81102270-6923-4e17-8da4-4b3829a3249b" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595506 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd187816-4859-45ca-a89b-831c75b4eed0" containerName="revoke-keys" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595511 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b7a44c-d384-4f50-a740-fffe46c2ed46" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595518 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595525 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62fcbd8-e151-4e4d-a525-8f08904e222c" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595532 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e109c52-c2fd-49b8-9172-374d5b43f3ea" containerName="istio-proxy" Oct 30 19:32:59.596696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.595539 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="342c0ddb-5be4-46be-8ee5-704bad5babf8" containerName="istio-proxy" Oct 30 19:32:59.598954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.598936 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.606015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.605568 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:32:59.606015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.605577 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:32:59.606015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.605667 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:32:59.606015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.605742 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:32:59.609988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.609962 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm"] Oct 30 19:32:59.637564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.637524 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.637769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.637693 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.637769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.637722 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654nk\" (UniqueName: \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.739509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.739472 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.739692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.739580 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.739692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.739603 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654nk\" (UniqueName: \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.740275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.740249 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.741868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.741840 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.747824 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.747799 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654nk\" (UniqueName: \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\") pod \"maas-api-revoke-sub-be0964eb-gpdkm\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:32:59.917219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:32:59.917136 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:33:00.279645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.279612 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm"] Oct 30 19:33:00.279902 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:33:00.279874 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b09d63d_85e2_4315_8c6c_4e7a59bfe4fd.slice/crio-c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930 WatchSource:0}: Error finding container c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930: Status 404 returned error can't find the container with id c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930 Oct 30 19:33:00.346988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.346940 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" event={"ID":"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd","Type":"ContainerStarted","Data":"c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930"} Oct 30 19:33:00.440618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.440587 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl"] Oct 30 19:33:00.446160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.446135 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.454362 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.454334 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl"] Oct 30 19:33:00.563001 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.562909 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.563001 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.562982 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.563251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.563114 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrtnk\" (UniqueName: \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.664923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.664885 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrtnk\" (UniqueName: \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.665339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.665061 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.665339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.665106 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.665752 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.665732 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.667493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.667468 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.674118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.674095 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrtnk\" (UniqueName: \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\") pod \"maas-api-revoke-sub-84159f60-8djkl\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:00.763575 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:00.763543 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:01.099759 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:33:01.099729 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3980a1f9_5137_4289_b097_2ec6f1023536.slice/crio-ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a WatchSource:0}: Error finding container ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a: Status 404 returned error can't find the container with id ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a Oct 30 19:33:01.100868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.100837 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl"] Oct 30 19:33:01.357275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.357241 6907 generic.go:356] "Generic (PLEG): container finished" podID="1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" containerID="8d3b193f4af4643729871dd224fa700fc1c8c6e698e3a8c820e5a9516d9986bf" exitCode=0 Oct 30 19:33:01.357432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.357324 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" event={"ID":"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd","Type":"ContainerDied","Data":"8d3b193f4af4643729871dd224fa700fc1c8c6e698e3a8c820e5a9516d9986bf"} Oct 30 19:33:01.358937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.358913 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" event={"ID":"3980a1f9-5137-4289-b097-2ec6f1023536","Type":"ContainerStarted","Data":"ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a"} Oct 30 19:33:01.636216 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.636154 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" containerID="cri-o://70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85" gracePeriod=25 Oct 30 19:33:01.643721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:01.643679 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" containerID="cri-o://c4d16586d8d2de64cc6fd0d230553c413151236663722db28a045d8e3d21848d" gracePeriod=25 Oct 30 19:33:02.369015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.368921 6907 generic.go:356] "Generic (PLEG): container finished" podID="3980a1f9-5137-4289-b097-2ec6f1023536" containerID="7f36faf402cecb428803a2a5f2315bb91253b8630ea16d03f8356657cadf667d" exitCode=0 Oct 30 19:33:02.369475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.369006 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" event={"ID":"3980a1f9-5137-4289-b097-2ec6f1023536","Type":"ContainerDied","Data":"7f36faf402cecb428803a2a5f2315bb91253b8630ea16d03f8356657cadf667d"} Oct 30 19:33:02.703692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.703647 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:33:02.714867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.714834 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\") pod \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " Oct 30 19:33:02.715075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.714915 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\" (UniqueName: \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\") pod \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " Oct 30 19:33:02.715141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.715118 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\") pod \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\" (UID: \"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd\") " Oct 30 19:33:02.715779 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.715742 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca" pod "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" (UID: "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:02.717996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.717957 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk" pod "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" (UID: "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd"). InnerVolumeSpecName "kube-api-access-654nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:02.718135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.718108 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca" pod "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" (UID: "1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:02.756507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.755520 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh"] Oct 30 19:33:02.766138 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.764946 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-639c15-w8vbh"] Oct 30 19:33:02.818818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.818783 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:02.818818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.818812 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:02.818818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:02.818824 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654nk\" (UniqueName: \"kubernetes.io/projected/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd-kube-api-access-654nk\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.198527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.196584 6907 patch_prober.go:29] interesting pod/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 19:33:03.198527 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.196645 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 19:33:03.292840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.292718 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:33:03.329743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.329179 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.329920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.329793 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.362163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.362133 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:33:03.397354 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.397313 6907 generic.go:356] "Generic (PLEG): container finished" podID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerID="b7fb7f26f2811b1c7dda37e1b4796090293b42051551224cd3cc4c8f1c99aa49" exitCode=0 Oct 30 19:33:03.397962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.397370 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" event={"ID":"b2cba290-51b0-46a3-8dd0-0472b0aab59e","Type":"ContainerDied","Data":"b7fb7f26f2811b1c7dda37e1b4796090293b42051551224cd3cc4c8f1c99aa49"} Oct 30 19:33:03.399565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.399539 6907 generic.go:356] "Generic (PLEG): container finished" podID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerID="663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872" exitCode=0 Oct 30 19:33:03.400162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.399597 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" Oct 30 19:33:03.400162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.399627 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" event={"ID":"b01ed4a7-3fe8-473e-97da-fa9231d4299c","Type":"ContainerDied","Data":"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872"} Oct 30 19:33:03.400162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.399979 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv" event={"ID":"b01ed4a7-3fe8-473e-97da-fa9231d4299c","Type":"ContainerDied","Data":"b53f6548c44a000bdd7d7e4ad42b67f699fd5fc21247d5759f0eedce9f56a7ac"} Oct 30 19:33:03.400162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.400017 6907 scope.go:122] "RemoveContainer" containerID="663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872" Oct 30 19:33:03.402634 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.402596 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" event={"ID":"1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd","Type":"ContainerDied","Data":"c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930"} Oct 30 19:33:03.402634 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.402614 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm" Oct 30 19:33:03.402965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.402639 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c727256e021027cf8f5177e4866f1027de27d5c923dcdecbad029038244f0930" Oct 30 19:33:03.405187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.405163 6907 generic.go:356] "Generic (PLEG): container finished" podID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerID="d4f344e11cb38fa5d78f717c752814572286fd1cc4abf901f3f75b3e425f2096" exitCode=0 Oct 30 19:33:03.405359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.405222 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" event={"ID":"58e39e7c-afbd-4f29-8387-83fb07385c2b","Type":"ContainerDied","Data":"d4f344e11cb38fa5d78f717c752814572286fd1cc4abf901f3f75b3e425f2096"} Oct 30 19:33:03.407323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.407302 6907 generic.go:356] "Generic (PLEG): container finished" podID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerID="1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010" exitCode=0 Oct 30 19:33:03.407409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.407372 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" Oct 30 19:33:03.407547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.407415 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerDied","Data":"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010"} Oct 30 19:33:03.407547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.407493 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7" event={"ID":"784acf91-2cc1-4789-949e-faf0ffd6a82b","Type":"ContainerDied","Data":"bb0bd7b41d38170c043c06301e39c1737bdcc136deab9c1cd2dce08275adfadd"} Oct 30 19:33:03.419679 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.419651 6907 scope.go:122] "RemoveContainer" containerID="663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872" Oct 30 19:33:03.422406 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:03.421190 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872\": container with ID starting with 663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872 not found: ID does not exist" containerID="663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872" Oct 30 19:33:03.422406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.421226 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872"} err="failed to get container status \"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872\": rpc error: code = NotFound desc = could not find container \"663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872\": container with ID starting with 663e7c603a81d3bf3b99d89a56eff06b49d113c3bcc57458f91eb24cb2a4f872 not found: ID does not exist" Oct 30 19:33:03.422406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.421252 6907 scope.go:122] "RemoveContainer" containerID="1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010" Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430453 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430575 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430623 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430661 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430725 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430757 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430856 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.430933 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\") pod \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\" (UID: \"b01ed4a7-3fe8-473e-97da-fa9231d4299c\") " Oct 30 19:33:03.431852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.431615 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.434336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.432578 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.434336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.433569 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.434850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.434705 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:03.434850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.434767 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.435636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.435605 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.437258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.437207 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "kube-api-access-fr94x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.441931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.441901 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.442027 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.442000 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo" pod "b01ed4a7-3fe8-473e-97da-fa9231d4299c" (UID: "b01ed4a7-3fe8-473e-97da-fa9231d4299c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:03.446866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.446763 6907 scope.go:122] "RemoveContainer" containerID="55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113" Oct 30 19:33:03.456794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.456379 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:33:03.460028 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.460007 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:33:03.464116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.464095 6907 scope.go:122] "RemoveContainer" containerID="1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010" Oct 30 19:33:03.464612 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:03.464586 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010\": container with ID starting with 1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010 not found: ID does not exist" containerID="1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010" Oct 30 19:33:03.464713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.464623 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010"} err="failed to get container status \"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010\": rpc error: code = NotFound desc = could not find container \"1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010\": container with ID starting with 1937e18bbb9d607fb73669c330abbf00de9116afa66056bca03fcbb0ab858010 not found: ID does not exist" Oct 30 19:33:03.464713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.464646 6907 scope.go:122] "RemoveContainer" containerID="55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113" Oct 30 19:33:03.464967 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:03.464946 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113\": container with ID starting with 55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113 not found: ID does not exist" containerID="55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113" Oct 30 19:33:03.465041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.464973 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113"} err="failed to get container status \"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113\": rpc error: code = NotFound desc = could not find container \"55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113\": container with ID starting with 55249a671ec9bb1fa83f356ff8c573e0d7a552afe41ca8c446adbdb53da8e113 not found: ID does not exist" Oct 30 19:33:03.527988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.527707 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" containerID="cri-o://a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406" gracePeriod=25 Oct 30 19:33:03.532928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.532897 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.532955 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.532993 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533049 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533089 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533244 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533322 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.533411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533385 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.534016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533496 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\") pod \"784acf91-2cc1-4789-949e-faf0ffd6a82b\" (UID: \"784acf91-2cc1-4789-949e-faf0ffd6a82b\") " Oct 30 19:33:03.534016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.533779 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.534179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534009 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.534331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534307 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534338 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr94x\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-kube-api-access-fr94x\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534354 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534370 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534383 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534397 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534410 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534426 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534459 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534474 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b01ed4a7-3fe8-473e-97da-fa9231d4299c-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.534747 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534683 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.535021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534869 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.535021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.534947 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:03.536317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.536278 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:03.536995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.536971 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.537676 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.537651 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.538347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.538316 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb" pod "784acf91-2cc1-4789-949e-faf0ffd6a82b" (UID: "784acf91-2cc1-4789-949e-faf0ffd6a82b"). InnerVolumeSpecName "kube-api-access-jjlhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.599238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.599215 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:03.635240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635178 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.635472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635274 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.635472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635318 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.635472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635351 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.635472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635386 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.635472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635463 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635499 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\") pod \"3980a1f9-5137-4289-b097-2ec6f1023536\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635526 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635563 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\" (UniqueName: \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\") pod \"3980a1f9-5137-4289-b097-2ec6f1023536\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635592 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635639 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635674 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635762 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635822 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635852 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\") pod \"3980a1f9-5137-4289-b097-2ec6f1023536\" (UID: \"3980a1f9-5137-4289-b097-2ec6f1023536\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635913 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635965 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635996 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636025 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636082 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\") pod \"58e39e7c-afbd-4f29-8387-83fb07385c2b\" (UID: \"58e39e7c-afbd-4f29-8387-83fb07385c2b\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636159 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\") pod \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\" (UID: \"b2cba290-51b0-46a3-8dd0-0472b0aab59e\") " Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.635814 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636796 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjlhb\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-kube-api-access-jjlhb\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636820 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636838 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636854 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636868 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636881 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636894 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/784acf91-2cc1-4789-949e-faf0ffd6a82b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636908 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/784acf91-2cc1-4789-949e-faf0ffd6a82b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.637251 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.637432 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.637641 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.637891 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.638113 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636182 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.636317 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca" pod "3980a1f9-5137-4289-b097-2ec6f1023536" (UID: "3980a1f9-5137-4289-b097-2ec6f1023536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.638303 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.638858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.638516 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.640836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640262 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.640836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640284 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.640836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640543 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.640836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640617 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "kube-api-access-zzlgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.640836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640719 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:03.641088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.640933 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.641221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.641194 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:03.641594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.641568 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo" pod "58e39e7c-afbd-4f29-8387-83fb07385c2b" (UID: "58e39e7c-afbd-4f29-8387-83fb07385c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:03.641674 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.641585 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p" pod "b2cba290-51b0-46a3-8dd0-0472b0aab59e" (UID: "b2cba290-51b0-46a3-8dd0-0472b0aab59e"). InnerVolumeSpecName "kube-api-access-2x76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.642749 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.642714 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca" pod "3980a1f9-5137-4289-b097-2ec6f1023536" (UID: "3980a1f9-5137-4289-b097-2ec6f1023536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:03.643043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.643017 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk" pod "3980a1f9-5137-4289-b097-2ec6f1023536" (UID: "3980a1f9-5137-4289-b097-2ec6f1023536"). InnerVolumeSpecName "kube-api-access-lrtnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:03.726195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.726163 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:33:03.732193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.732163 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-j6rgv"] Oct 30 19:33:03.738423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738399 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738424 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738435 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrtnk\" (UniqueName: \"kubernetes.io/projected/3980a1f9-5137-4289-b097-2ec6f1023536-kube-api-access-lrtnk\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738463 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738473 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738482 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738490 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738499 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738507 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3980a1f9-5137-4289-b097-2ec6f1023536-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738517 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738525 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzlgw\" (UniqueName: \"kubernetes.io/projected/58e39e7c-afbd-4f29-8387-83fb07385c2b-kube-api-access-zzlgw\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738534 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738543 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738551 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738560 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738568 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58e39e7c-afbd-4f29-8387-83fb07385c2b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738600 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x76p\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-kube-api-access-2x76p\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738613 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738624 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2cba290-51b0-46a3-8dd0-0472b0aab59e-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.738680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.738633 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58e39e7c-afbd-4f29-8387-83fb07385c2b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:03.747182 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.747152 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:33:03.761113 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:03.761084 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-jdcn7"] Oct 30 19:33:04.161597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.161563 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222ccb1b-fc39-43a9-8943-b9d96d794512" path="/var/lib/kubelet/pods/222ccb1b-fc39-43a9-8943-b9d96d794512/volumes" Oct 30 19:33:04.162049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.162035 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" path="/var/lib/kubelet/pods/784acf91-2cc1-4789-949e-faf0ffd6a82b/volumes" Oct 30 19:33:04.162543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.162495 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" path="/var/lib/kubelet/pods/b01ed4a7-3fe8-473e-97da-fa9231d4299c/volumes" Oct 30 19:33:04.416978 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.416885 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" event={"ID":"b2cba290-51b0-46a3-8dd0-0472b0aab59e","Type":"ContainerDied","Data":"18d661a6ce5ea461ebcb9f6827e484ea804d724d3218ba765aa0e01ac6414ca7"} Oct 30 19:33:04.416978 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.416957 6907 scope.go:122] "RemoveContainer" containerID="b7fb7f26f2811b1c7dda37e1b4796090293b42051551224cd3cc4c8f1c99aa49" Oct 30 19:33:04.417417 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.416905 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp" Oct 30 19:33:04.419878 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.419834 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" Oct 30 19:33:04.419995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.419875 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl" event={"ID":"3980a1f9-5137-4289-b097-2ec6f1023536","Type":"ContainerDied","Data":"ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a"} Oct 30 19:33:04.419995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.419937 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4e34fd8de6fdb12aa04febfa531baf80340238335f3d792d82131c884d033a" Oct 30 19:33:04.421847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.421828 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" Oct 30 19:33:04.421956 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.421894 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s" event={"ID":"58e39e7c-afbd-4f29-8387-83fb07385c2b","Type":"ContainerDied","Data":"6526d985e0ad7131785c12c8c6a5eff92cfc13f2c08b3ab49478f36ceb807b4d"} Oct 30 19:33:04.428942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.428924 6907 scope.go:122] "RemoveContainer" containerID="d4f344e11cb38fa5d78f717c752814572286fd1cc4abf901f3f75b3e425f2096" Oct 30 19:33:04.438846 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.438820 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:33:04.443275 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.443255 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-mtxpp"] Oct 30 19:33:04.458488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.458464 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:33:04.463097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:04.463069 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-171808-openshift-default-856d98c98c-rmx9s"] Oct 30 19:33:05.919785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:05.919741 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:06.151466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:06.151415 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" path="/var/lib/kubelet/pods/58e39e7c-afbd-4f29-8387-83fb07385c2b/volumes" Oct 30 19:33:06.151958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:06.151944 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" path="/var/lib/kubelet/pods/b2cba290-51b0-46a3-8dd0-0472b0aab59e/volumes" Oct 30 19:33:07.740043 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:07.740002 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:33:07.937422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:07.937384 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:08.195335 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.195226 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" containerID="cri-o://e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7" gracePeriod=45 Oct 30 19:33:08.519007 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.518975 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:33:08.529318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.529281 6907 generic.go:356] "Generic (PLEG): container finished" podID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerID="e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7" exitCode=0 Oct 30 19:33:08.529555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.529321 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" Oct 30 19:33:08.529555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.529363 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" event={"ID":"b0c9ffb0-c8ec-4663-9880-745dabf09139","Type":"ContainerDied","Data":"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7"} Oct 30 19:33:08.529555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.529417 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" event={"ID":"b0c9ffb0-c8ec-4663-9880-745dabf09139","Type":"ContainerDied","Data":"66a74f500a3620fb88bc461e8b3c4e42df29b26eb599cd3d47ba03e28f371e1b"} Oct 30 19:33:08.529555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.529438 6907 scope.go:122] "RemoveContainer" containerID="e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7" Oct 30 19:33:08.549657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.549632 6907 scope.go:122] "RemoveContainer" containerID="e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7" Oct 30 19:33:08.550085 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:08.550052 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7\": container with ID starting with e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7 not found: ID does not exist" containerID="e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7" Oct 30 19:33:08.550315 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.550284 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7"} err="failed to get container status \"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7\": rpc error: code = NotFound desc = could not find container \"e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7\": container with ID starting with e868a214e6c4ecf5c9971bc19f2f109b7da3b1ab12bf0a5baff4500cb5f1c4d7 not found: ID does not exist" Oct 30 19:33:08.657582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657546 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.657582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657595 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.657849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657662 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.657849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657741 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.657849 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657781 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\" (UniqueName: \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.658003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657854 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\") pod \"b0c9ffb0-c8ec-4663-9880-745dabf09139\" (UID: \"b0c9ffb0-c8ec-4663-9880-745dabf09139\") " Oct 30 19:33:08.658003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.657938 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:08.658122 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.658055 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:08.658255 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.658223 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:08.658426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.658411 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.658556 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.658428 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.658556 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.658439 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.660099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.660074 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:08.660233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.660215 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:08.660286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.660249 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd" pod "b0c9ffb0-c8ec-4663-9880-745dabf09139" (UID: "b0c9ffb0-c8ec-4663-9880-745dabf09139"). InnerVolumeSpecName "kube-api-access-79gfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:08.759543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.759423 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0c9ffb0-c8ec-4663-9880-745dabf09139-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.759543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.759489 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0c9ffb0-c8ec-4663-9880-745dabf09139-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.759543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.759501 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gfd\" (UniqueName: \"kubernetes.io/projected/b0c9ffb0-c8ec-4663-9880-745dabf09139-kube-api-access-79gfd\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:08.857331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.857299 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:33:08.861342 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:08.861314 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt"] Oct 30 19:33:09.436035 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:09.435991 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6cf459/distinct-w1-6cf459-kserve-59f664d5d7-rqbmt" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:09.670761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:09.670722 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:33:09.989046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:09.989013 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:10.153388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:10.153351 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" path="/var/lib/kubelet/pods/b0c9ffb0-c8ec-4663-9880-745dabf09139/volumes" Oct 30 19:33:11.010470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:11.008598 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:11.852279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:11.842675 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw"] Oct 30 19:33:12.156566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.156097 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f7f075-qwmjw"] Oct 30 19:33:12.436523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.434963 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd187816-4859-45ca-a89b-831c75b4eed0" path="/var/lib/kubelet/pods/dd187816-4859-45ca-a89b-831c75b4eed0/volumes" Oct 30 19:33:12.771641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.770668 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:33:12.771641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.770933 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" containerID="cri-o://2fad8aea7b8f5081a83bb8792d5facb3f019fd1f9c0e6ea00c4139387797c361" gracePeriod=30 Oct 30 19:33:12.799883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.794518 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:33:12.799883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:12.794771 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" containerID="cri-o://b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015" gracePeriod=30 Oct 30 19:33:13.069528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.068519 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:33:13.069528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.068839 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" containerID="cri-o://4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261" gracePeriod=30 Oct 30 19:33:13.715159 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.714635 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:13.715159 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.714728 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:13.960465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.957842 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:33:13.960465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:13.957942 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:33:15.714717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.714689 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:33:15.824042 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.823936 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\" (UniqueName: \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\") pod \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " Oct 30 19:33:15.824228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.824046 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\") pod \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\" (UID: \"4fad9cf2-0d81-4dcf-9216-c85a3e32476b\") " Oct 30 19:33:15.830532 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.830218 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume" pod "4fad9cf2-0d81-4dcf-9216-c85a3e32476b" (UID: "4fad9cf2-0d81-4dcf-9216-c85a3e32476b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:15.836610 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.836577 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq" pod "4fad9cf2-0d81-4dcf-9216-c85a3e32476b" (UID: "4fad9cf2-0d81-4dcf-9216-c85a3e32476b"). InnerVolumeSpecName "kube-api-access-hmxlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:15.920926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.920884 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:15.925620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.925591 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:15.925620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.925618 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxlq\" (UniqueName: \"kubernetes.io/projected/4fad9cf2-0d81-4dcf-9216-c85a3e32476b-kube-api-access-hmxlq\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:15.950392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.950362 6907 generic.go:356] "Generic (PLEG): container finished" podID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerID="10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe" exitCode=0 Oct 30 19:33:15.950589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.950410 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" Oct 30 19:33:15.950589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.950418 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" event={"ID":"4fad9cf2-0d81-4dcf-9216-c85a3e32476b","Type":"ContainerDied","Data":"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe"} Oct 30 19:33:15.950589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.950485 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7" event={"ID":"4fad9cf2-0d81-4dcf-9216-c85a3e32476b","Type":"ContainerDied","Data":"db116370a3c5ec66abf363874c85162778a895043c5330a51927ca18cdcdd8e6"} Oct 30 19:33:15.950589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.950518 6907 scope.go:122] "RemoveContainer" containerID="10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe" Oct 30 19:33:15.962094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.962076 6907 scope.go:122] "RemoveContainer" containerID="10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe" Oct 30 19:33:15.962382 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:15.962365 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe\": container with ID starting with 10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe not found: ID does not exist" containerID="10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe" Oct 30 19:33:15.962439 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.962389 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe"} err="failed to get container status \"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe\": rpc error: code = NotFound desc = could not find container \"10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe\": container with ID starting with 10bae1673bee92ffbd3e7cf05ed4739e672f9b4fab9c44a69a30014b58ba0cfe not found: ID does not exist" Oct 30 19:33:15.976173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.976140 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:33:15.981361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:15.981331 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-639c15-7d546bdf65-6n4j7"] Oct 30 19:33:16.151265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:16.151182 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" path="/var/lib/kubelet/pods/4fad9cf2-0d81-4dcf-9216-c85a3e32476b/volumes" Oct 30 19:33:17.880269 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:17.880234 6907 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e697cc8_ad70_4d97_b1da_905022976883.slice/crio-b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:33:18.032507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.031562 6907 generic.go:356] "Generic (PLEG): container finished" podID="8e697cc8-ad70-4d97-b1da-905022976883" containerID="b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015" exitCode=0 Oct 30 19:33:18.032507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.031806 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" event={"ID":"8e697cc8-ad70-4d97-b1da-905022976883","Type":"ContainerDied","Data":"b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015"} Oct 30 19:33:18.035046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.033710 6907 generic.go:356] "Generic (PLEG): container finished" podID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerID="2fad8aea7b8f5081a83bb8792d5facb3f019fd1f9c0e6ea00c4139387797c361" exitCode=0 Oct 30 19:33:18.035046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.033789 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" event={"ID":"09b83a86-3762-4388-ae94-ae19a3dbabe2","Type":"ContainerDied","Data":"2fad8aea7b8f5081a83bb8792d5facb3f019fd1f9c0e6ea00c4139387797c361"} Oct 30 19:33:18.078097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.078027 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:33:18.117599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117571 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117630 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117653 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117680 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117702 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117767 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117823 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.117995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117883 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.117995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.117952 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.118054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118000 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\") pod \"8e697cc8-ad70-4d97-b1da-905022976883\" (UID: \"8e697cc8-ad70-4d97-b1da-905022976883\") " Oct 30 19:33:18.118054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118009 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.118117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118092 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.118548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118526 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.118653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118550 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.118653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118571 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.118899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.118866 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.119435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.119410 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:18.120044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.120019 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.121006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.120984 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.121603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.121572 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:18.122039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.122012 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv" pod "8e697cc8-ad70-4d97-b1da-905022976883" (UID: "8e697cc8-ad70-4d97-b1da-905022976883"). InnerVolumeSpecName "kube-api-access-qfcdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.169903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.169876 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:33:18.219150 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219124 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219175 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219270 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219308 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219339 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219431 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219490 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219533 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.219699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219574 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219611 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\") pod \"09b83a86-3762-4388-ae94-ae19a3dbabe2\" (UID: \"09b83a86-3762-4388-ae94-ae19a3dbabe2\") " Oct 30 19:33:18.219858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219746 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.219946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.219926 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.220155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220135 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220162 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220176 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220191 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220206 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e697cc8-ad70-4d97-b1da-905022976883-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220219 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e697cc8-ad70-4d97-b1da-905022976883-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220234 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e697cc8-ad70-4d97-b1da-905022976883-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220249 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220265 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfcdv\" (UniqueName: \"kubernetes.io/projected/8e697cc8-ad70-4d97-b1da-905022976883-kube-api-access-qfcdv\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.220520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220217 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.220520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.220355 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:18.222319 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.222289 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:18.222997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.222970 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.223090 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.222979 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "kube-api-access-d8tk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.223293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.223274 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token" pod "09b83a86-3762-4388-ae94-ae19a3dbabe2" (UID: "09b83a86-3762-4388-ae94-ae19a3dbabe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.321425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321383 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tk2\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-kube-api-access-d8tk2\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.321425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321427 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.321688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321499 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09b83a86-3762-4388-ae94-ae19a3dbabe2-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.321688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321518 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.321688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321532 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09b83a86-3762-4388-ae94-ae19a3dbabe2-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.321688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.321546 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09b83a86-3762-4388-ae94-ae19a3dbabe2-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.354473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.354427 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:33:18.422429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422393 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422673 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422481 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422673 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422539 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422673 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422637 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422695 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422728 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.422857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422798 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.423033 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422867 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.423033 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422909 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\") pod \"390ac4fa-5635-41aa-8817-52c771435684\" (UID: \"390ac4fa-5635-41aa-8817-52c771435684\") " Oct 30 19:33:18.423296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.422862 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.423296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423074 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.423296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423262 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.423659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423552 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:18.423659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423586 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.423659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423609 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.423659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423626 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.423659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.423633 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.424830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.424809 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.425234 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.425218 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "kube-api-access-nt69r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:18.425491 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.425464 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:18.425599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.425494 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo" pod "390ac4fa-5635-41aa-8817-52c771435684" (UID: "390ac4fa-5635-41aa-8817-52c771435684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:18.525504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525429 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/390ac4fa-5635-41aa-8817-52c771435684-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.525504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525508 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt69r\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-kube-api-access-nt69r\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.525716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525526 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.525716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525543 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/390ac4fa-5635-41aa-8817-52c771435684-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.525716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525557 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/390ac4fa-5635-41aa-8817-52c771435684-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:18.525716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:18.525570 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/390ac4fa-5635-41aa-8817-52c771435684-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:19.043269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.043237 6907 generic.go:356] "Generic (PLEG): container finished" podID="390ac4fa-5635-41aa-8817-52c771435684" containerID="4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261" exitCode=0 Oct 30 19:33:19.043766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.043285 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" Oct 30 19:33:19.043766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.043319 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" event={"ID":"390ac4fa-5635-41aa-8817-52c771435684","Type":"ContainerDied","Data":"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261"} Oct 30 19:33:19.043766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.043380 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g" event={"ID":"390ac4fa-5635-41aa-8817-52c771435684","Type":"ContainerDied","Data":"b92ac3a044519e658ffeef1a39a31d8deabfe53d223110c62ba5045a2c522757"} Oct 30 19:33:19.043766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.043395 6907 scope.go:122] "RemoveContainer" containerID="4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261" Oct 30 19:33:19.045269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.045243 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" Oct 30 19:33:19.045385 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.045279 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4" event={"ID":"09b83a86-3762-4388-ae94-ae19a3dbabe2","Type":"ContainerDied","Data":"ad8c2608adc03adc23e6751681614759a4611abe99aeb4dd9cc08aaf5fea0946"} Oct 30 19:33:19.047197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.047180 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" Oct 30 19:33:19.047304 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.047250 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7" event={"ID":"8e697cc8-ad70-4d97-b1da-905022976883","Type":"ContainerDied","Data":"43479c5f10aa2963244ddd03fe0e27f55efae7ea9e3b5a0a135664dd3832ae41"} Oct 30 19:33:19.057425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.057407 6907 scope.go:122] "RemoveContainer" containerID="4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261" Oct 30 19:33:19.057707 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:19.057689 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261\": container with ID starting with 4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261 not found: ID does not exist" containerID="4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261" Oct 30 19:33:19.057780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.057715 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261"} err="failed to get container status \"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261\": rpc error: code = NotFound desc = could not find container \"4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261\": container with ID starting with 4855c0c8e4bca5677a8c31a119697a4a3c8569d72ae9e1083eb1490d3370b261 not found: ID does not exist" Oct 30 19:33:19.057780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.057731 6907 scope.go:122] "RemoveContainer" containerID="2fad8aea7b8f5081a83bb8792d5facb3f019fd1f9c0e6ea00c4139387797c361" Oct 30 19:33:19.070364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.070339 6907 scope.go:122] "RemoveContainer" containerID="b2bd90714c5e636b69b0e4f236a965bd1d7ca40ec15fd052ccabf77ae285d015" Oct 30 19:33:19.074768 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.074733 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:33:19.083937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.083913 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kskp7"] Oct 30 19:33:19.095809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.095752 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:33:19.102974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.102948 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-kl89g"] Oct 30 19:33:19.116404 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.116374 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:33:19.127428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.123090 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-639c15-openshift-default-58d59f577f-c8qv4"] Oct 30 19:33:19.148788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.148753 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:33:19.989760 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:19.989724 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:20.150925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:20.150886 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" path="/var/lib/kubelet/pods/09b83a86-3762-4388-ae94-ae19a3dbabe2/volumes" Oct 30 19:33:20.151403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:20.151389 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="390ac4fa-5635-41aa-8817-52c771435684" path="/var/lib/kubelet/pods/390ac4fa-5635-41aa-8817-52c771435684/volumes" Oct 30 19:33:20.151868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:20.151855 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e697cc8-ad70-4d97-b1da-905022976883" path="/var/lib/kubelet/pods/8e697cc8-ad70-4d97-b1da-905022976883/volumes" Oct 30 19:33:20.987461 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:20.987402 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:21.713427 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.713389 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:33:21.713853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.713595 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" containerID="cri-o://169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba" gracePeriod=30 Oct 30 19:33:21.724963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.724931 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:33:21.725137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.725116 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" containerID="cri-o://9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d" gracePeriod=30 Oct 30 19:33:21.748761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.748716 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:33:21.749261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.749181 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" containerID="cri-o://45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7" gracePeriod=30 Oct 30 19:33:21.754510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.754416 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:33:21.754706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:21.754664 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" containerID="cri-o://2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914" gracePeriod=30 Oct 30 19:33:22.045549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.045522 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:33:22.080425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.080389 6907 generic.go:356] "Generic (PLEG): container finished" podID="45daa901-2222-4562-82f0-e0752975b69f" containerID="c4d16586d8d2de64cc6fd0d230553c413151236663722db28a045d8e3d21848d" exitCode=0 Oct 30 19:33:22.080573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.080501 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" event={"ID":"45daa901-2222-4562-82f0-e0752975b69f","Type":"ContainerDied","Data":"c4d16586d8d2de64cc6fd0d230553c413151236663722db28a045d8e3d21848d"} Oct 30 19:33:22.080573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.080563 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" event={"ID":"45daa901-2222-4562-82f0-e0752975b69f","Type":"ContainerDied","Data":"8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf"} Oct 30 19:33:22.080703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.080587 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab89d85a3d4c2ead948915f841cfa7371bbed70254ed31f3e99e875d5e71aaf" Oct 30 19:33:22.082745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.082713 6907 generic.go:356] "Generic (PLEG): container finished" podID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerID="70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85" exitCode=0 Oct 30 19:33:22.082937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.082772 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" Oct 30 19:33:22.082937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.082825 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" event={"ID":"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93","Type":"ContainerDied","Data":"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85"} Oct 30 19:33:22.082937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.082865 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp" event={"ID":"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93","Type":"ContainerDied","Data":"ccc34dcba6574bf0387137f8666da3b1ec99ec8e09839ebc78eca63f7bb17f27"} Oct 30 19:33:22.082937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.082890 6907 scope.go:122] "RemoveContainer" containerID="70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85" Oct 30 19:33:22.092625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.092596 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\") pod \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " Oct 30 19:33:22.092833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.092815 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\" (UniqueName: \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\") pod \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\" (UID: \"e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93\") " Oct 30 19:33:22.093241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.092990 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume" pod "e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" (UID: "e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:22.093367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.093338 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:22.095868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.095841 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8" pod "e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" (UID: "e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93"). InnerVolumeSpecName "kube-api-access-bwkc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:22.097200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.097178 6907 scope.go:122] "RemoveContainer" containerID="70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85" Oct 30 19:33:22.097556 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:22.097534 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85\": container with ID starting with 70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85 not found: ID does not exist" containerID="70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85" Oct 30 19:33:22.097645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.097568 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85"} err="failed to get container status \"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85\": rpc error: code = NotFound desc = could not find container \"70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85\": container with ID starting with 70d09b0d3d23b09eb33bc91083633ecccc4db0f2f8c6a6969e4c663974d24a85 not found: ID does not exist" Oct 30 19:33:22.102632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.102613 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:33:22.214326 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.214237 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\" (UniqueName: \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\") pod \"45daa901-2222-4562-82f0-e0752975b69f\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " Oct 30 19:33:22.214326 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.214323 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\") pod \"45daa901-2222-4562-82f0-e0752975b69f\" (UID: \"45daa901-2222-4562-82f0-e0752975b69f\") " Oct 30 19:33:22.215261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.215231 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwkc8\" (UniqueName: \"kubernetes.io/projected/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93-kube-api-access-bwkc8\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:22.219697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.219652 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume" pod "45daa901-2222-4562-82f0-e0752975b69f" (UID: "45daa901-2222-4562-82f0-e0752975b69f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:22.233820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.233780 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s" pod "45daa901-2222-4562-82f0-e0752975b69f" (UID: "45daa901-2222-4562-82f0-e0752975b69f"). InnerVolumeSpecName "kube-api-access-n4q5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:22.316629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.316595 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4q5s\" (UniqueName: \"kubernetes.io/projected/45daa901-2222-4562-82f0-e0752975b69f-kube-api-access-n4q5s\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:22.316629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.316622 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45daa901-2222-4562-82f0-e0752975b69f-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:22.406910 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.406873 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:33:22.410611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.410587 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-f7f075-6c86dc5759-qgqnp"] Oct 30 19:33:22.518263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.518218 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:33:22.518461 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:22.518285 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:23.093044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.093016 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz" Oct 30 19:33:23.127570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.127484 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:33:23.134680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.134651 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-f7f075-7ddb645bc5-4m2dz"] Oct 30 19:33:23.854766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.854736 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:33:23.941401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.941362 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\") pod \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " Oct 30 19:33:23.941599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.941550 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\" (UniqueName: \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\") pod \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\" (UID: \"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f\") " Oct 30 19:33:23.944058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.944022 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume" pod "db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" (UID: "db9ddcad-b84c-48eb-bf61-92a4dd1bf91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:23.954910 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.954867 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:33:23.956391 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.956357 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" containerID="cri-o://2afc535e158f631c56ac25a92fd5169ae137977999e497617420f282b029c61d" gracePeriod=30 Oct 30 19:33:23.956905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.956877 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq" pod "db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" (UID: "db9ddcad-b84c-48eb-bf61-92a4dd1bf91f"). InnerVolumeSpecName "kube-api-access-b9drq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:23.972812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.972769 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:33:23.973759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.973727 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" containerID="cri-o://a7f239150a327ed82aca0b256d2e51e6bdd0c117b42947fd8e23dbe7a2ec51e6" gracePeriod=30 Oct 30 19:33:23.977631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.977610 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:33:23.977795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.977778 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" containerID="cri-o://f9f76ad8151216d095083691c0d3f97bc4d2063d7361590c8aa801dd5e5841f0" gracePeriod=30 Oct 30 19:33:23.986566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.986540 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:33:23.986741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:23.986722 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" containerID="cri-o://fa02f84a4f4ae53727dc94b5952ec763d12e3a28b7b0e28178d759711e04404c" gracePeriod=30 Oct 30 19:33:24.043632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.043598 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9drq\" (UniqueName: \"kubernetes.io/projected/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-kube-api-access-b9drq\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:24.043632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.043632 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:24.179286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.179186 6907 generic.go:356] "Generic (PLEG): container finished" podID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerID="a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406" exitCode=0 Oct 30 19:33:24.179727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.179317 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" Oct 30 19:33:24.181381 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.181358 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45daa901-2222-4562-82f0-e0752975b69f" path="/var/lib/kubelet/pods/45daa901-2222-4562-82f0-e0752975b69f/volumes" Oct 30 19:33:24.181876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.181861 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" path="/var/lib/kubelet/pods/e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93/volumes" Oct 30 19:33:24.182302 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.182286 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" event={"ID":"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f","Type":"ContainerDied","Data":"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406"} Oct 30 19:33:24.182366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.182323 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6" event={"ID":"db9ddcad-b84c-48eb-bf61-92a4dd1bf91f","Type":"ContainerDied","Data":"1d411e7962ec79a46f22a114477ab58c775939b1cc0ada8586240f92fd15b915"} Oct 30 19:33:24.182366 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.182348 6907 scope.go:122] "RemoveContainer" containerID="a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406" Oct 30 19:33:24.193486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.193432 6907 scope.go:122] "RemoveContainer" containerID="a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406" Oct 30 19:33:24.193854 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:24.193835 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406\": container with ID starting with a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406 not found: ID does not exist" containerID="a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406" Oct 30 19:33:24.193910 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.193862 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406"} err="failed to get container status \"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406\": rpc error: code = NotFound desc = could not find container \"a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406\": container with ID starting with a0f643588084efe64fd6e5484daec0379ab96eabe73910791b8bd4ec7c7c9406 not found: ID does not exist" Oct 30 19:33:24.306496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.306458 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:33:24.310121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.310094 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6cf459-64ffb9669f-rpsn6"] Oct 30 19:33:24.538244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.538200 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:33:24.785756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.785702 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn"] Oct 30 19:33:24.787104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787073 6907 cpu_manager.go:414] "containerMap: removing container" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" Oct 30 19:33:24.787104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787101 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787129 6907 cpu_manager.go:414] "containerMap: removing container" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787138 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787150 6907 cpu_manager.go:414] "containerMap: removing container" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787158 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787169 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787176 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787198 6907 cpu_manager.go:414] "containerMap: removing container" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787206 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787215 6907 cpu_manager.go:414] "containerMap: removing container" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787223 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787272 6907 cpu_manager.go:414] "containerMap: removing container" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787279 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787293 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" Oct 30 19:33:24.787311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787299 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787327 6907 cpu_manager.go:414] "containerMap: removing container" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787334 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787380 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787386 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787400 6907 cpu_manager.go:414] "containerMap: removing container" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787407 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787420 6907 cpu_manager.go:414] "containerMap: removing container" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787426 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787468 6907 cpu_manager.go:414] "containerMap: removing container" podUID="1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787475 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787504 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787511 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787529 6907 cpu_manager.go:414] "containerMap: removing container" podUID="3980a1f9-5137-4289-b097-2ec6f1023536" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787536 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3980a1f9-5137-4289-b097-2ec6f1023536" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787792 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b01ed4a7-3fe8-473e-97da-fa9231d4299c" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787805 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fad9cf2-0d81-4dcf-9216-c85a3e32476b" containerName="payload-processing" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787820 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787829 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787849 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e697cc8-ad70-4d97-b1da-905022976883" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787866 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="3980a1f9-5137-4289-b097-2ec6f1023536" containerName="revoke-keys" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787874 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="390ac4fa-5635-41aa-8817-52c771435684" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787888 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b83a86-3762-4388-ae94-ae19a3dbabe2" containerName="istio-proxy" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787905 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="45daa901-2222-4562-82f0-e0752975b69f" containerName="payload-pre-processing" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787921 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c9ffb0-c8ec-4663-9880-745dabf09139" containerName="main" Oct 30 19:33:24.787922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787931 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" containerName="payload-pre-processing" Oct 30 19:33:24.789059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787946 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6bb7e43-bcc2-46c7-92ba-f0f63b9c1e93" containerName="payload-processing" Oct 30 19:33:24.789059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787965 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="784acf91-2cc1-4789-949e-faf0ffd6a82b" containerName="istio-proxy" Oct 30 19:33:24.789059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787975 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cba290-51b0-46a3-8dd0-0472b0aab59e" containerName="istio-proxy" Oct 30 19:33:24.789059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.787993 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e39e7c-afbd-4f29-8387-83fb07385c2b" containerName="istio-proxy" Oct 30 19:33:24.794850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.794824 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.799504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.799439 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:33:24.799656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.799532 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:33:24.799656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.799570 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:33:24.799656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.799620 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:33:24.807312 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.807276 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn"] Oct 30 19:33:24.867347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.867298 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.867561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.867389 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.867561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.867424 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.972182 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.972133 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.972400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.972220 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.972400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.972251 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.973191 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.973160 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.975189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.975165 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:24.989090 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:24.989062 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\") pod \"maas-api-revoke-sub-578da981-58zdn\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:25.067279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:25.067189 6907 patch_prober.go:29] interesting pod/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:33:25.067279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:25.067263 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:25.153346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:25.153308 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:25.443510 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:33:25.443478 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6701363_dcd9_484f_9f6e_aa7d40db8774.slice/crio-e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539 WatchSource:0}: Error finding container e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539: Status 404 returned error can't find the container with id e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539 Oct 30 19:33:25.444350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:25.444299 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn"] Oct 30 19:33:26.150580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:26.150489 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9ddcad-b84c-48eb-bf61-92a4dd1bf91f" path="/var/lib/kubelet/pods/db9ddcad-b84c-48eb-bf61-92a4dd1bf91f/volumes" Oct 30 19:33:26.199317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:26.199279 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" event={"ID":"d6701363-dcd9-484f-9f6e-aa7d40db8774","Type":"ContainerStarted","Data":"9a0d6d3ca9e7d1336c63db06be60a648d27a813d636f02b2a211047953472cfa"} Oct 30 19:33:26.199509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:26.199343 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" event={"ID":"d6701363-dcd9-484f-9f6e-aa7d40db8774","Type":"ContainerStarted","Data":"e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539"} Oct 30 19:33:26.217462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:26.217353 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" podStartSLOduration=2.217336485 podStartE2EDuration="2.217336485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:26.217297339 +0000 UTC m=+812.193523283" watchObservedRunningTime="2026-10-30 19:33:26.217336485 +0000 UTC m=+812.193562432" Oct 30 19:33:27.059868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.059171 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:33:27.098737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.098710 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:33:27.127884 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.127840 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:33:27.133700 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.133674 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:33:27.206085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.205878 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.206085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.205943 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.206085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206004 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.206085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206034 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.206085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206065 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206096 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206124 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206151 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206181 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206221 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.206541 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.206265 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.207877 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208098 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208213 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208208 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208265 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208316 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208401 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208705 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208778 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208818 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208847 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208875 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208913 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208949 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\") pod \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\" (UID: \"56b04c47-f047-4cc6-9ad1-2ca346155ec0\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.208981 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209019 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209078 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209114 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209173 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209216 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209294 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209342 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209401 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209437 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209553 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.210052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.209596 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.212629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.211232 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.212629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.211296 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\") pod \"9232aebc-4241-415f-90d5-822695de5791\" (UID: \"9232aebc-4241-415f-90d5-822695de5791\") " Oct 30 19:33:27.212629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.212307 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\") pod \"72faf153-3f73-433a-9122-a3167466366c\" (UID: \"72faf153-3f73-433a-9122-a3167466366c\") " Oct 30 19:33:27.212629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.212386 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\") pod \"8a404769-ddd9-473d-8d14-bd668ced40bd\" (UID: \"8a404769-ddd9-473d-8d14-bd668ced40bd\") " Oct 30 19:33:27.213345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213321 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.213435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213321 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.213435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213351 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "kube-api-access-lzcxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.213435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213392 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.213651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213436 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "kube-api-access-5nkbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.214108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.213886 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:27.214546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.214243 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.215755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.215367 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.215755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.215394 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a404769-ddd9-473d-8d14-bd668ced40bd-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.215755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.215408 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.215755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.215426 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216011 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216043 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216060 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216083 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216099 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzcxf\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-kube-api-access-lzcxf\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216117 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216132 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216152 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkbf\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-kube-api-access-5nkbf\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216162 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216412 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.216694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.216628 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.217634 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.217354 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.218073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.218047 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.218654 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.218627 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "kube-api-access-8gcx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.218874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.218851 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.219074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.219050 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:27.220899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.220841 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.221098 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221077 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.221322 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221271 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.221403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221337 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.221566 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221532 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.221848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221823 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.221944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.221847 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.224188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.224122 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.224691 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.224433 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "kube-api-access-76cs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.224858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.224829 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:27.225170 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.225141 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert" pod "72faf153-3f73-433a-9122-a3167466366c" (UID: "72faf153-3f73-433a-9122-a3167466366c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:27.225722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.225693 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:27.227970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.227941 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:27.227970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.227938 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo" pod "56b04c47-f047-4cc6-9ad1-2ca346155ec0" (UID: "56b04c47-f047-4cc6-9ad1-2ca346155ec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:27.228159 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.228135 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token" pod "9232aebc-4241-415f-90d5-822695de5791" (UID: "9232aebc-4241-415f-90d5-822695de5791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:27.238503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.235881 6907 generic.go:356] "Generic (PLEG): container finished" podID="d6701363-dcd9-484f-9f6e-aa7d40db8774" containerID="9a0d6d3ca9e7d1336c63db06be60a648d27a813d636f02b2a211047953472cfa" exitCode=0 Oct 30 19:33:27.238503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.236016 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" event={"ID":"d6701363-dcd9-484f-9f6e-aa7d40db8774","Type":"ContainerDied","Data":"9a0d6d3ca9e7d1336c63db06be60a648d27a813d636f02b2a211047953472cfa"} Oct 30 19:33:27.238503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.238004 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo" pod "8a404769-ddd9-473d-8d14-bd668ced40bd" (UID: "8a404769-ddd9-473d-8d14-bd668ced40bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:27.242561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.242493 6907 generic.go:356] "Generic (PLEG): container finished" podID="9232aebc-4241-415f-90d5-822695de5791" containerID="9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d" exitCode=0 Oct 30 19:33:27.242690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.242630 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" event={"ID":"9232aebc-4241-415f-90d5-822695de5791","Type":"ContainerDied","Data":"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d"} Oct 30 19:33:27.242690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.242638 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" Oct 30 19:33:27.242690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.242682 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj" event={"ID":"9232aebc-4241-415f-90d5-822695de5791","Type":"ContainerDied","Data":"6257f60bfab48d2f67dc980549044a34c616c49015a2b3d5475cb55d77f793fe"} Oct 30 19:33:27.242861 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.242713 6907 scope.go:122] "RemoveContainer" containerID="9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d" Oct 30 19:33:27.245394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.245370 6907 generic.go:356] "Generic (PLEG): container finished" podID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerID="45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7" exitCode=0 Oct 30 19:33:27.245552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.245534 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" Oct 30 19:33:27.247715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.247686 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" event={"ID":"56b04c47-f047-4cc6-9ad1-2ca346155ec0","Type":"ContainerDied","Data":"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7"} Oct 30 19:33:27.247830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.247749 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq" event={"ID":"56b04c47-f047-4cc6-9ad1-2ca346155ec0","Type":"ContainerDied","Data":"769ee81a5bdc1bfd4a3d29809f9727086557edc509b181ed181eb29175dd70e6"} Oct 30 19:33:27.253339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.253315 6907 generic.go:356] "Generic (PLEG): container finished" podID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerID="169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba" exitCode=0 Oct 30 19:33:27.253515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.253414 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" event={"ID":"8a404769-ddd9-473d-8d14-bd668ced40bd","Type":"ContainerDied","Data":"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba"} Oct 30 19:33:27.253515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.253478 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" event={"ID":"8a404769-ddd9-473d-8d14-bd668ced40bd","Type":"ContainerDied","Data":"aa9693fc408d88b1418c500204b083be0cb3b3aa72c035b613229141a403ed23"} Oct 30 19:33:27.253649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.253514 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx" Oct 30 19:33:27.255802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.255779 6907 generic.go:356] "Generic (PLEG): container finished" podID="72faf153-3f73-433a-9122-a3167466366c" containerID="2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914" exitCode=0 Oct 30 19:33:27.255974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.255838 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" Oct 30 19:33:27.256085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.255842 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" event={"ID":"72faf153-3f73-433a-9122-a3167466366c","Type":"ContainerDied","Data":"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914"} Oct 30 19:33:27.256150 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.256108 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm" event={"ID":"72faf153-3f73-433a-9122-a3167466366c","Type":"ContainerDied","Data":"eac69892b830fdf036e3e2f57ea084485a7076b65bbebd7ad4c0b2d573725337"} Oct 30 19:33:27.261924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.261899 6907 scope.go:122] "RemoveContainer" containerID="9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d" Oct 30 19:33:27.262257 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:27.262230 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d\": container with ID starting with 9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d not found: ID does not exist" containerID="9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d" Oct 30 19:33:27.262409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.262377 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d"} err="failed to get container status \"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d\": rpc error: code = NotFound desc = could not find container \"9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d\": container with ID starting with 9cab767e93f8b6143da2fc21564a05493d23c5b0486027ad26d45e3b833dd48d not found: ID does not exist" Oct 30 19:33:27.262409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.262410 6907 scope.go:122] "RemoveContainer" containerID="45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7" Oct 30 19:33:27.279810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.279786 6907 scope.go:122] "RemoveContainer" containerID="45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7" Oct 30 19:33:27.280214 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:27.280189 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7\": container with ID starting with 45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7 not found: ID does not exist" containerID="45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7" Oct 30 19:33:27.280278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.280223 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7"} err="failed to get container status \"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7\": rpc error: code = NotFound desc = could not find container \"45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7\": container with ID starting with 45c2f662f5e8d239763228615a6937888109aaf4a4bcc1429bae0e21bd4652d7 not found: ID does not exist" Oct 30 19:33:27.280278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.280242 6907 scope.go:122] "RemoveContainer" containerID="169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba" Oct 30 19:33:27.291960 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.291934 6907 scope.go:122] "RemoveContainer" containerID="169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba" Oct 30 19:33:27.292603 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:27.292579 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba\": container with ID starting with 169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba not found: ID does not exist" containerID="169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba" Oct 30 19:33:27.292773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.292706 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba"} err="failed to get container status \"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba\": rpc error: code = NotFound desc = could not find container \"169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba\": container with ID starting with 169595babd20ad89efd86f381daff9f7457f0df56894d8375fdfc28f66e78bba not found: ID does not exist" Oct 30 19:33:27.292773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.292740 6907 scope.go:122] "RemoveContainer" containerID="2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914" Oct 30 19:33:27.307244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.307223 6907 scope.go:122] "RemoveContainer" containerID="2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914" Oct 30 19:33:27.307583 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:27.307560 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914\": container with ID starting with 2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914 not found: ID does not exist" containerID="2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914" Oct 30 19:33:27.307688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.307589 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914"} err="failed to get container status \"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914\": rpc error: code = NotFound desc = could not find container \"2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914\": container with ID starting with 2d1d4b22b6162dbb708f69ec45a97be12182ff020023cbe438c680cc3bfa0914 not found: ID does not exist" Oct 30 19:33:27.318578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318556 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318707 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318691 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318713 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72faf153-3f73-433a-9122-a3167466366c-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318727 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318741 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318755 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.318780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318769 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9232aebc-4241-415f-90d5-822695de5791-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318785 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318800 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318813 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318827 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318841 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56b04c47-f047-4cc6-9ad1-2ca346155ec0-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318856 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cs7\" (UniqueName: \"kubernetes.io/projected/8a404769-ddd9-473d-8d14-bd668ced40bd-kube-api-access-76cs7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.318870 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319190 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72faf153-3f73-433a-9122-a3167466366c-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319212 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9232aebc-4241-415f-90d5-822695de5791-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319229 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcx5\" (UniqueName: \"kubernetes.io/projected/72faf153-3f73-433a-9122-a3167466366c-kube-api-access-8gcx5\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319244 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319259 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319274 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a404769-ddd9-473d-8d14-bd668ced40bd-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319289 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72faf153-3f73-433a-9122-a3167466366c-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319303 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9232aebc-4241-415f-90d5-822695de5791-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319319 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a404769-ddd9-473d-8d14-bd668ced40bd-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.319376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.319335 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9232aebc-4241-415f-90d5-822695de5791-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:27.322414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.322393 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:33:27.326095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.326076 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-j77pj"] Oct 30 19:33:27.362086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.362056 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:33:27.367627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.367595 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-nfbfq"] Oct 30 19:33:27.382961 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.382928 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:33:27.386609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.386582 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-g76bm"] Oct 30 19:33:27.405013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.404984 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:33:27.410569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.410539 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f7f075-openshift-default-7f77b5b897-vpvdx"] Oct 30 19:33:27.603266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.603178 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 19:33:27.603266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.603237 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:27.899197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.899112 6907 patch_prober.go:29] interesting pod/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 19:33:27.899197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:27.899182 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:28.150976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.150900 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" path="/var/lib/kubelet/pods/56b04c47-f047-4cc6-9ad1-2ca346155ec0/volumes" Oct 30 19:33:28.151378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.151364 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72faf153-3f73-433a-9122-a3167466366c" path="/var/lib/kubelet/pods/72faf153-3f73-433a-9122-a3167466366c/volumes" Oct 30 19:33:28.151799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.151787 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" path="/var/lib/kubelet/pods/8a404769-ddd9-473d-8d14-bd668ced40bd/volumes" Oct 30 19:33:28.152533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.152519 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9232aebc-4241-415f-90d5-822695de5791" path="/var/lib/kubelet/pods/9232aebc-4241-415f-90d5-822695de5791/volumes" Oct 30 19:33:28.518030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.517997 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:28.634276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.634224 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\") pod \"d6701363-dcd9-484f-9f6e-aa7d40db8774\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " Oct 30 19:33:28.634509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.634427 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\") pod \"d6701363-dcd9-484f-9f6e-aa7d40db8774\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " Oct 30 19:33:28.634509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.634488 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\") pod \"d6701363-dcd9-484f-9f6e-aa7d40db8774\" (UID: \"d6701363-dcd9-484f-9f6e-aa7d40db8774\") " Oct 30 19:33:28.634709 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.634684 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca" pod "d6701363-dcd9-484f-9f6e-aa7d40db8774" (UID: "d6701363-dcd9-484f-9f6e-aa7d40db8774"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:28.634847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.634831 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:28.636680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.636650 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca" pod "d6701363-dcd9-484f-9f6e-aa7d40db8774" (UID: "d6701363-dcd9-484f-9f6e-aa7d40db8774"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:28.636680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.636671 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8" pod "d6701363-dcd9-484f-9f6e-aa7d40db8774" (UID: "d6701363-dcd9-484f-9f6e-aa7d40db8774"). InnerVolumeSpecName "kube-api-access-5hnf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:28.735617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.735586 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6701363-dcd9-484f-9f6e-aa7d40db8774-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:28.735782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:28.735621 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnf8\" (UniqueName: \"kubernetes.io/projected/d6701363-dcd9-484f-9f6e-aa7d40db8774-kube-api-access-5hnf8\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.293553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.293476 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" event={"ID":"d6701363-dcd9-484f-9f6e-aa7d40db8774","Type":"ContainerDied","Data":"e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539"} Oct 30 19:33:29.293553 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.293539 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5145f33c434bc9031dc0275133593a842ec9d09684aa0038f07fcfdf5eb9539" Oct 30 19:33:29.293842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.293621 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn" Oct 30 19:33:29.297901 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.297867 6907 generic.go:356] "Generic (PLEG): container finished" podID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerID="f9f76ad8151216d095083691c0d3f97bc4d2063d7361590c8aa801dd5e5841f0" exitCode=0 Oct 30 19:33:29.298571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.298033 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" event={"ID":"ebefcc59-4d77-49f1-8c01-00792e94fe3b","Type":"ContainerDied","Data":"f9f76ad8151216d095083691c0d3f97bc4d2063d7361590c8aa801dd5e5841f0"} Oct 30 19:33:29.307284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.305758 6907 generic.go:356] "Generic (PLEG): container finished" podID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerID="2afc535e158f631c56ac25a92fd5169ae137977999e497617420f282b029c61d" exitCode=0 Oct 30 19:33:29.307284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.305907 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" event={"ID":"04e9ff52-ea3d-4372-b16b-7e6739738397","Type":"ContainerDied","Data":"2afc535e158f631c56ac25a92fd5169ae137977999e497617420f282b029c61d"} Oct 30 19:33:29.309477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.308543 6907 generic.go:356] "Generic (PLEG): container finished" podID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerID="fa02f84a4f4ae53727dc94b5952ec763d12e3a28b7b0e28178d759711e04404c" exitCode=0 Oct 30 19:33:29.309477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.308645 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" event={"ID":"956c90bd-7b71-44b1-9444-65dfe74e159b","Type":"ContainerDied","Data":"fa02f84a4f4ae53727dc94b5952ec763d12e3a28b7b0e28178d759711e04404c"} Oct 30 19:33:29.312611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.311498 6907 generic.go:356] "Generic (PLEG): container finished" podID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerID="a7f239150a327ed82aca0b256d2e51e6bdd0c117b42947fd8e23dbe7a2ec51e6" exitCode=0 Oct 30 19:33:29.312611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.311626 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" event={"ID":"4987b25d-8d31-4a79-a263-673e137e0bd6","Type":"ContainerDied","Data":"a7f239150a327ed82aca0b256d2e51e6bdd0c117b42947fd8e23dbe7a2ec51e6"} Oct 30 19:33:29.317402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.317062 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:33:29.323810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.323054 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:33:29.370033 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.370003 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:33:29.375378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.375352 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:33:29.444624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.444589 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.444818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.444655 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.444818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.444711 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.445077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445050 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.445235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445218 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.445385 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445369 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.445536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445519 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.445651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445628 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445736 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445750 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445811 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445865 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445911 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445943 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.445986 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446088 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446196 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446307 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446015 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446545 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446600 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446645 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\") pod \"04e9ff52-ea3d-4372-b16b-7e6739738397\" (UID: \"04e9ff52-ea3d-4372-b16b-7e6739738397\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446676 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446709 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\") pod \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\" (UID: \"ebefcc59-4d77-49f1-8c01-00792e94fe3b\") " Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.446862 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447237 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447261 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447275 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447318 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447332 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447347 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447360 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447458 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:29.447540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447517 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.449050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447626 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "kube-api-access-grrmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.449050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447731 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.449050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.447885 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.449050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.448629 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.449258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.449070 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.449258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.449152 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "kube-api-access-6hhs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.449410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.449387 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.449977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.449958 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo" pod "ebefcc59-4d77-49f1-8c01-00792e94fe3b" (UID: "ebefcc59-4d77-49f1-8c01-00792e94fe3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:29.451024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.450994 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo" pod "04e9ff52-ea3d-4372-b16b-7e6739738397" (UID: "04e9ff52-ea3d-4372-b16b-7e6739738397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:29.551680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.542996 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" containerID="cri-o://81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf" gracePeriod=25 Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.549899 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554778 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554845 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554895 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554935 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554994 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555041 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555081 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555125 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555186 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555225 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555287 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555352 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555394 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.555438 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556247 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\") pod \"4987b25d-8d31-4a79-a263-673e137e0bd6\" (UID: \"4987b25d-8d31-4a79-a263-673e137e0bd6\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556283 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.556468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556316 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\") pod \"956c90bd-7b71-44b1-9444-65dfe74e159b\" (UID: \"956c90bd-7b71-44b1-9444-65dfe74e159b\") " Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556877 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556900 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e9ff52-ea3d-4372-b16b-7e6739738397-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556915 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556930 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556946 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556960 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrmb\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-kube-api-access-grrmb\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556976 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hhs9\" (UniqueName: \"kubernetes.io/projected/04e9ff52-ea3d-4372-b16b-7e6739738397-kube-api-access-6hhs9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.556991 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.557004 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.557022 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e9ff52-ea3d-4372-b16b-7e6739738397-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.557480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.557037 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebefcc59-4d77-49f1-8c01-00792e94fe3b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.554668 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.560543 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.560712 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.565732 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.566030 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.567873 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.568374 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.568727 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.569607 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "kube-api-access-dbng6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.569845 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.570237 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.571484 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "kube-api-access-nwb8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.573377 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.574351 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:29.575683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.574488 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:29.577534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.577097 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token" pod "956c90bd-7b71-44b1-9444-65dfe74e159b" (UID: "956c90bd-7b71-44b1-9444-65dfe74e159b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.582600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.581884 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:29.585067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.583685 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token" pod "4987b25d-8d31-4a79-a263-673e137e0bd6" (UID: "4987b25d-8d31-4a79-a263-673e137e0bd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:29.658645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658613 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4987b25d-8d31-4a79-a263-673e137e0bd6-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658641 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658651 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658659 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658668 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658681 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwb8n\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-kube-api-access-nwb8n\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658695 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658703 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658711 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658719 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658728 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658738 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbng6\" (UniqueName: \"kubernetes.io/projected/4987b25d-8d31-4a79-a263-673e137e0bd6-kube-api-access-dbng6\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658746 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658756 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956c90bd-7b71-44b1-9444-65dfe74e159b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658769 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658779 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4987b25d-8d31-4a79-a263-673e137e0bd6-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658787 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:29.658918 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:29.658795 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956c90bd-7b71-44b1-9444-65dfe74e159b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:30.327844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.326604 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" event={"ID":"ebefcc59-4d77-49f1-8c01-00792e94fe3b","Type":"ContainerDied","Data":"3fc0a422b1f78ba1b0193bd7a51ee666d3f2fea45ebe2fdf0dcbc816a40ce4ac"} Oct 30 19:33:30.327844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.326688 6907 scope.go:122] "RemoveContainer" containerID="f9f76ad8151216d095083691c0d3f97bc4d2063d7361590c8aa801dd5e5841f0" Oct 30 19:33:30.327844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.326867 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd" Oct 30 19:33:30.334056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.334031 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" event={"ID":"04e9ff52-ea3d-4372-b16b-7e6739738397","Type":"ContainerDied","Data":"372a534927dc32b876b9f8f078366e1a2e56486d055e417404ea456f86a723d3"} Oct 30 19:33:30.334313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.334298 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l" Oct 30 19:33:30.352522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.352339 6907 scope.go:122] "RemoveContainer" containerID="2afc535e158f631c56ac25a92fd5169ae137977999e497617420f282b029c61d" Oct 30 19:33:30.356922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.356187 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" event={"ID":"956c90bd-7b71-44b1-9444-65dfe74e159b","Type":"ContainerDied","Data":"5c7781f456563409d36495cf805301bce1fcf7ed5347b185a45bbdaa87914812"} Oct 30 19:33:30.356922 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.356325 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr" Oct 30 19:33:30.364347 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.364314 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" event={"ID":"4987b25d-8d31-4a79-a263-673e137e0bd6","Type":"ContainerDied","Data":"9f8d93dc1405e0cb508c39a08f451a4bb3dabe85bd8d90b6f4a2b434e0815f21"} Oct 30 19:33:30.364603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.364588 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz" Oct 30 19:33:30.374374 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.374344 6907 scope.go:122] "RemoveContainer" containerID="fa02f84a4f4ae53727dc94b5952ec763d12e3a28b7b0e28178d759711e04404c" Oct 30 19:33:30.377439 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.377418 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:33:30.381772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.381749 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-lwsmd"] Oct 30 19:33:30.385670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.385651 6907 scope.go:122] "RemoveContainer" containerID="a7f239150a327ed82aca0b256d2e51e6bdd0c117b42947fd8e23dbe7a2ec51e6" Oct 30 19:33:30.393102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.393073 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:33:30.400188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.400166 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-t7djr"] Oct 30 19:33:30.433671 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.433635 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:33:30.437477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.437431 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-mz58l"] Oct 30 19:33:30.445083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.445043 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:33:30.448789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:30.448767 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6cf459-openshift-default-5cb8bc465c-h67rz"] Oct 30 19:33:32.150913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:32.150882 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" path="/var/lib/kubelet/pods/04e9ff52-ea3d-4372-b16b-7e6739738397/volumes" Oct 30 19:33:32.151359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:32.151345 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" path="/var/lib/kubelet/pods/4987b25d-8d31-4a79-a263-673e137e0bd6/volumes" Oct 30 19:33:32.151769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:32.151756 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" path="/var/lib/kubelet/pods/956c90bd-7b71-44b1-9444-65dfe74e159b/volumes" Oct 30 19:33:32.152514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:32.152499 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" path="/var/lib/kubelet/pods/ebefcc59-4d77-49f1-8c01-00792e94fe3b/volumes" Oct 30 19:33:34.170324 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.170283 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" containerID="cri-o://073b8158edec3c5e6b890361a90bc4c56c7d8a1249ad1bb248465d44c1f2a7db" gracePeriod=45 Oct 30 19:33:34.405237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.405207 6907 generic.go:356] "Generic (PLEG): container finished" podID="8e40913e-6543-401e-b90d-402694f7dc1e" containerID="073b8158edec3c5e6b890361a90bc4c56c7d8a1249ad1bb248465d44c1f2a7db" exitCode=0 Oct 30 19:33:34.405388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.405290 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" event={"ID":"8e40913e-6543-401e-b90d-402694f7dc1e","Type":"ContainerDied","Data":"073b8158edec3c5e6b890361a90bc4c56c7d8a1249ad1bb248465d44c1f2a7db"} Oct 30 19:33:34.453113 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.453088 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:33:34.524169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524125 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.524830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524288 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.524830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524351 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\" (UniqueName: \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.524830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524397 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.524830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524425 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.524830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.524558 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\") pod \"8e40913e-6543-401e-b90d-402694f7dc1e\" (UID: \"8e40913e-6543-401e-b90d-402694f7dc1e\") " Oct 30 19:33:34.525831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.525733 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:34.525831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.525806 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:34.526213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.526192 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:34.527032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.526996 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:34.527696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.527671 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:33:34.527876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.527838 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt" pod "8e40913e-6543-401e-b90d-402694f7dc1e" (UID: "8e40913e-6543-401e-b90d-402694f7dc1e"). InnerVolumeSpecName "kube-api-access-729jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:34.626647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626616 6907 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-tmp-dir\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.626647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626644 6907 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-home\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.626647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626652 6907 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e40913e-6543-401e-b90d-402694f7dc1e-tls-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.626883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626660 6907 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-dshm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.626883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626671 6907 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e40913e-6543-401e-b90d-402694f7dc1e-model-cache\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.626883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.626680 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729jt\" (UniqueName: \"kubernetes.io/projected/8e40913e-6543-401e-b90d-402694f7dc1e-kube-api-access-729jt\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:34.912011 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:34.911971 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:35.415385 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:35.415349 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" Oct 30 19:33:35.415862 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:35.415350 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn" event={"ID":"8e40913e-6543-401e-b90d-402694f7dc1e","Type":"ContainerDied","Data":"967e675eaf702a72953d9de97d92f52f515384e6f3016e32e3393afe06a4435e"} Oct 30 19:33:35.415862 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:35.415498 6907 scope.go:122] "RemoveContainer" containerID="073b8158edec3c5e6b890361a90bc4c56c7d8a1249ad1bb248465d44c1f2a7db" Oct 30 19:33:35.442896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:35.442867 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:33:35.447318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:35.447289 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3bc085/distinct-w4-3bc085-kserve-7bbbd56fc6-dc8nn"] Oct 30 19:33:36.150266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:36.150187 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" path="/var/lib/kubelet/pods/8e40913e-6543-401e-b90d-402694f7dc1e/volumes" Oct 30 19:33:37.736216 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:37.736173 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:33:37.913774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:37.913744 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:33:44.911490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:44.911435 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:33:45.351413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:45.351380 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:33:47.268585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.268546 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:33:47.269097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.268818 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" containerID="cri-o://8c5da4c5407caa38ad42dd63f79758c5d36b81d556275e54fdbfb446540ff9d5" gracePeriod=30 Oct 30 19:33:47.272677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.272642 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:33:47.272887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.272864 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" containerID="cri-o://a227ce00920c2cb7e5ebba32fb111491c31fb1a854ae66882ba7eba6cd838a38" gracePeriod=30 Oct 30 19:33:47.276213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.276189 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:33:47.276375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.276357 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" containerID="cri-o://2d0a62ac1f0ebd9b390af02dc6cf2f4f0f1da83653665edd22fbee6ba4391509" gracePeriod=30 Oct 30 19:33:47.279313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.279288 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:33:47.279519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:47.279484 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" containerID="cri-o://3d6c7096815444383927ac19bb876def50dd9eb9599a7db6a93a8d6e61bae26a" gracePeriod=30 Oct 30 19:33:49.833786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:49.833726 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:33:49.933519 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:49.933483 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\") pod \"1be55339-a855-4dab-89a2-2abac9065974\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " Oct 30 19:33:49.933706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:49.933545 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\" (UniqueName: \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\") pod \"1be55339-a855-4dab-89a2-2abac9065974\" (UID: \"1be55339-a855-4dab-89a2-2abac9065974\") " Oct 30 19:33:49.934001 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:49.933970 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume" pod "1be55339-a855-4dab-89a2-2abac9065974" (UID: "1be55339-a855-4dab-89a2-2abac9065974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:49.935726 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:49.935702 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9" pod "1be55339-a855-4dab-89a2-2abac9065974" (UID: "1be55339-a855-4dab-89a2-2abac9065974"). InnerVolumeSpecName "kube-api-access-8r4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:50.035273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.035236 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be55339-a855-4dab-89a2-2abac9065974-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:50.035273 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.035268 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4r9\" (UniqueName: \"kubernetes.io/projected/1be55339-a855-4dab-89a2-2abac9065974-kube-api-access-8r4r9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:50.586908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.586873 6907 generic.go:356] "Generic (PLEG): container finished" podID="1be55339-a855-4dab-89a2-2abac9065974" containerID="81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf" exitCode=0 Oct 30 19:33:50.587092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.586921 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" Oct 30 19:33:50.587092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.586951 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" event={"ID":"1be55339-a855-4dab-89a2-2abac9065974","Type":"ContainerDied","Data":"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf"} Oct 30 19:33:50.587092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.587005 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k" event={"ID":"1be55339-a855-4dab-89a2-2abac9065974","Type":"ContainerDied","Data":"913a67e4a009f6647b1f441d2216b33d269ab5017f82965a088d9361101e7610"} Oct 30 19:33:50.587092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.587027 6907 scope.go:122] "RemoveContainer" containerID="81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf" Oct 30 19:33:50.597990 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.597969 6907 scope.go:122] "RemoveContainer" containerID="81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf" Oct 30 19:33:50.598296 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:33:50.598276 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf\": container with ID starting with 81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf not found: ID does not exist" containerID="81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf" Oct 30 19:33:50.598340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.598306 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf"} err="failed to get container status \"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf\": rpc error: code = NotFound desc = could not find container \"81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf\": container with ID starting with 81a5eaa8525ae0b4c6b2101d16a15cbecfe5b3f565259cf0662854280ea5b9bf not found: ID does not exist" Oct 30 19:33:50.607893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.607866 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:33:50.617213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:50.617175 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3bc085-6ff798fd6c-gqc7k"] Oct 30 19:33:51.018597 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.018559 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:33:51.019083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019069 6907 cpu_manager.go:414] "containerMap: removing container" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" Oct 30 19:33:51.019127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019084 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" Oct 30 19:33:51.019127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019098 6907 cpu_manager.go:414] "containerMap: removing container" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" Oct 30 19:33:51.019127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019103 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" Oct 30 19:33:51.019127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019118 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" Oct 30 19:33:51.019127 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019123 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019134 6907 cpu_manager.go:414] "containerMap: removing container" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019139 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019151 6907 cpu_manager.go:414] "containerMap: removing container" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019155 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019166 6907 cpu_manager.go:414] "containerMap: removing container" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019171 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019177 6907 cpu_manager.go:414] "containerMap: removing container" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019181 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019191 6907 cpu_manager.go:414] "containerMap: removing container" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019195 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019206 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019211 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019218 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d6701363-dcd9-484f-9f6e-aa7d40db8774" containerName="revoke-keys" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019222 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6701363-dcd9-484f-9f6e-aa7d40db8774" containerName="revoke-keys" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019230 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" Oct 30 19:33:51.019279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019234 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019306 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="72faf153-3f73-433a-9122-a3167466366c" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019316 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="9232aebc-4241-415f-90d5-822695de5791" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019322 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e9ff52-ea3d-4372-b16b-7e6739738397" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019329 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e40913e-6543-401e-b90d-402694f7dc1e" containerName="main" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019335 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6701363-dcd9-484f-9f6e-aa7d40db8774" containerName="revoke-keys" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019340 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a404769-ddd9-473d-8d14-bd668ced40bd" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019347 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="956c90bd-7b71-44b1-9444-65dfe74e159b" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019352 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="4987b25d-8d31-4a79-a263-673e137e0bd6" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019358 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebefcc59-4d77-49f1-8c01-00792e94fe3b" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019364 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b04c47-f047-4cc6-9ad1-2ca346155ec0" containerName="istio-proxy" Oct 30 19:33:51.019783 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.019370 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be55339-a855-4dab-89a2-2abac9065974" containerName="payload-pre-processing" Oct 30 19:33:51.024219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.024198 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.033997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.033970 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:33:51.034121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.034091 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l2kjc\"" Oct 30 19:33:51.034226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.034198 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:33:51.034413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.034335 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:33:51.038018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.037992 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:33:51.148068 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.148023 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnsk\" (UniqueName: \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.148306 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.148290 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.148379 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.148336 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.249467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.249413 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnsk\" (UniqueName: \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.249711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.249676 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.249794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.249733 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.253220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.253186 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.253379 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.253221 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.259714 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.259681 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnsk\" (UniqueName: \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\") pod \"maas-controller-78455b774c-n2bt8\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.333944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.333836 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:33:51.333944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.333913 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:33:51.340485 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.338466 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:51.663632 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:33:51.662909 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9184944_65a5_4f9a_812e_d5d29a743897.slice/crio-28eabec80fdbdff51193103168798e763a00c0e2e0513494d33c48fa39fa55b8 WatchSource:0}: Error finding container 28eabec80fdbdff51193103168798e763a00c0e2e0513494d33c48fa39fa55b8: Status 404 returned error can't find the container with id 28eabec80fdbdff51193103168798e763a00c0e2e0513494d33c48fa39fa55b8 Oct 30 19:33:51.664617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.664586 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:33:51.664826 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:51.664805 6907 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:33:52.182424 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.181839 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be55339-a855-4dab-89a2-2abac9065974" path="/var/lib/kubelet/pods/1be55339-a855-4dab-89a2-2abac9065974/volumes" Oct 30 19:33:52.648702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.648290 6907 generic.go:356] "Generic (PLEG): container finished" podID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerID="3d6c7096815444383927ac19bb876def50dd9eb9599a7db6a93a8d6e61bae26a" exitCode=0 Oct 30 19:33:52.648702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.648502 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerDied","Data":"3d6c7096815444383927ac19bb876def50dd9eb9599a7db6a93a8d6e61bae26a"} Oct 30 19:33:52.648702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.648569 6907 scope.go:122] "RemoveContainer" containerID="ae4e54917fb5fe236a247d119acf87a97978c5f76168acd5ef05eb283dad8c9d" Oct 30 19:33:52.663651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.663532 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-n2bt8" event={"ID":"e9184944-65a5-4f9a-812e-d5d29a743897","Type":"ContainerStarted","Data":"28eabec80fdbdff51193103168798e763a00c0e2e0513494d33c48fa39fa55b8"} Oct 30 19:33:52.666636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.666460 6907 generic.go:356] "Generic (PLEG): container finished" podID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerID="8c5da4c5407caa38ad42dd63f79758c5d36b81d556275e54fdbfb446540ff9d5" exitCode=0 Oct 30 19:33:52.666636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.666551 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" event={"ID":"e8330f71-b86b-46fe-bb16-9e7c979308a8","Type":"ContainerDied","Data":"8c5da4c5407caa38ad42dd63f79758c5d36b81d556275e54fdbfb446540ff9d5"} Oct 30 19:33:52.678515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.678301 6907 generic.go:356] "Generic (PLEG): container finished" podID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerID="2d0a62ac1f0ebd9b390af02dc6cf2f4f0f1da83653665edd22fbee6ba4391509" exitCode=0 Oct 30 19:33:52.678515 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.678408 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" event={"ID":"60a7879a-cccd-4d31-b5f2-d56cafd48616","Type":"ContainerDied","Data":"2d0a62ac1f0ebd9b390af02dc6cf2f4f0f1da83653665edd22fbee6ba4391509"} Oct 30 19:33:52.697621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.697346 6907 generic.go:356] "Generic (PLEG): container finished" podID="a56c4323-7158-4170-9d3a-81256d394f1a" containerID="a227ce00920c2cb7e5ebba32fb111491c31fb1a854ae66882ba7eba6cd838a38" exitCode=0 Oct 30 19:33:52.697621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.697517 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" event={"ID":"a56c4323-7158-4170-9d3a-81256d394f1a","Type":"ContainerDied","Data":"a227ce00920c2cb7e5ebba32fb111491c31fb1a854ae66882ba7eba6cd838a38"} Oct 30 19:33:52.902403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.902031 6907 patch_prober.go:29] interesting pod/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:33:52.902403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:52.902094 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 19:33:54.076421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.076387 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:33:54.082387 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.081684 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:33:54.235545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.235515 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:33:54.255822 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.255735 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:33:54.278500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278454 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278502 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278558 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278576 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278603 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278643 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278661 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278688 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278749 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278776 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278802 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278826 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278847 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278877 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278896 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278915 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\") pod \"60a7879a-cccd-4d31-b5f2-d56cafd48616\" (UID: \"60a7879a-cccd-4d31-b5f2-d56cafd48616\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278947 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.278977 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.278975 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\") pod \"a56c4323-7158-4170-9d3a-81256d394f1a\" (UID: \"a56c4323-7158-4170-9d3a-81256d394f1a\") " Oct 30 19:33:54.279754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.279729 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.279914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.279890 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.280076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.280062 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.280299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.280282 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:54.281725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.281487 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.281725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.281720 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.282142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.281966 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.282142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.282063 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:54.282694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.282409 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.282694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.282409 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.282874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.282776 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.282874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.282844 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:54.284969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.284913 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:54.285398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.285370 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.285844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.285812 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.285941 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.285881 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs" pod "60a7879a-cccd-4d31-b5f2-d56cafd48616" (UID: "60a7879a-cccd-4d31-b5f2-d56cafd48616"). InnerVolumeSpecName "kube-api-access-g7scs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.287128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.287106 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "kube-api-access-7z6g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.287586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.287567 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy" pod "a56c4323-7158-4170-9d3a-81256d394f1a" (UID: "a56c4323-7158-4170-9d3a-81256d394f1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.379821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.379780 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.379862 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.379904 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.379948 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.379984 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380014 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380063 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380124 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380154 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380179 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380225 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380249 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380308 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380296 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380331 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380385 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380387 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380422 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380487 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380518 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\") pod \"cf26f935-3edf-458a-a38a-7a0ce517d86b\" (UID: \"cf26f935-3edf-458a-a38a-7a0ce517d86b\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380563 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\") pod \"e8330f71-b86b-46fe-bb16-9e7c979308a8\" (UID: \"e8330f71-b86b-46fe-bb16-9e7c979308a8\") " Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380646 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.380701 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.380670 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.381173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381115 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381138 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381154 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381170 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381185 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381199 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381213 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a56c4323-7158-4170-9d3a-81256d394f1a-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.381564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381432 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381654 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60a7879a-cccd-4d31-b5f2-d56cafd48616-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381874 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381894 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381910 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381924 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381939 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7scs\" (UniqueName: \"kubernetes.io/projected/60a7879a-cccd-4d31-b5f2-d56cafd48616-kube-api-access-g7scs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381955 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z6g9\" (UniqueName: \"kubernetes.io/projected/a56c4323-7158-4170-9d3a-81256d394f1a-kube-api-access-7z6g9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381970 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381984 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.381998 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382012 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382027 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a56c4323-7158-4170-9d3a-81256d394f1a-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382033 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382042 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a56c4323-7158-4170-9d3a-81256d394f1a-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382081 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60a7879a-cccd-4d31-b5f2-d56cafd48616-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382379 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382418 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382439 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382598 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.382659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.382629 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:33:54.383842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.383814 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:54.384540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.384492 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:33:54.384540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.384514 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.384711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.384531 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "kube-api-access-8n5lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.384711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.384569 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "kube-api-access-x6sbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.385078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.385049 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.385560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.385537 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy" pod "cf26f935-3edf-458a-a38a-7a0ce517d86b" (UID: "cf26f935-3edf-458a-a38a-7a0ce517d86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:33:54.385837 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.385814 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token" pod "e8330f71-b86b-46fe-bb16-9e7c979308a8" (UID: "e8330f71-b86b-46fe-bb16-9e7c979308a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:33:54.483725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483685 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483726 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483744 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n5lh\" (UniqueName: \"kubernetes.io/projected/cf26f935-3edf-458a-a38a-7a0ce517d86b-kube-api-access-8n5lh\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483762 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483776 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483790 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483804 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8330f71-b86b-46fe-bb16-9e7c979308a8-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483819 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf26f935-3edf-458a-a38a-7a0ce517d86b-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483832 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483848 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483862 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8330f71-b86b-46fe-bb16-9e7c979308a8-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483876 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6sbw\" (UniqueName: \"kubernetes.io/projected/e8330f71-b86b-46fe-bb16-9e7c979308a8-kube-api-access-x6sbw\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483890 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8330f71-b86b-46fe-bb16-9e7c979308a8-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:54.483944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:54.483904 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf26f935-3edf-458a-a38a-7a0ce517d86b-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:33:55.057189 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.057156 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" Oct 30 19:33:55.057376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.057154 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx" event={"ID":"e8330f71-b86b-46fe-bb16-9e7c979308a8","Type":"ContainerDied","Data":"bc627ab4a3b2d6f2f7ae8b99de3adccae42fdc8ac135cf27511f1462fdcac2a7"} Oct 30 19:33:55.057376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.057315 6907 scope.go:122] "RemoveContainer" containerID="8c5da4c5407caa38ad42dd63f79758c5d36b81d556275e54fdbfb446540ff9d5" Oct 30 19:33:55.059729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.059696 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" event={"ID":"60a7879a-cccd-4d31-b5f2-d56cafd48616","Type":"ContainerDied","Data":"798abc37fa76b93f435e1a341b2298431a5a9abc6a3943590d5cde79b5f4a3ba"} Oct 30 19:33:55.059872 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.059798 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc" Oct 30 19:33:55.068573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.068539 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" event={"ID":"a56c4323-7158-4170-9d3a-81256d394f1a","Type":"ContainerDied","Data":"1d29c3496a52b898f532c29d21a69e8ad5ba8edbbed356e531bfcf1b3a9c9a15"} Oct 30 19:33:55.068573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.068565 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t" Oct 30 19:33:55.076643 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.071435 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" event={"ID":"cf26f935-3edf-458a-a38a-7a0ce517d86b","Type":"ContainerDied","Data":"e6c4f417c88193b080fb5c5a5df6aa224fe414c678914eb95913aafac9948f4f"} Oct 30 19:33:55.076643 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.071480 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk" Oct 30 19:33:55.076643 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.072687 6907 scope.go:122] "RemoveContainer" containerID="2d0a62ac1f0ebd9b390af02dc6cf2f4f0f1da83653665edd22fbee6ba4391509" Oct 30 19:33:55.087692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.085831 6907 scope.go:122] "RemoveContainer" containerID="a227ce00920c2cb7e5ebba32fb111491c31fb1a854ae66882ba7eba6cd838a38" Oct 30 19:33:55.099760 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.099733 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:33:55.104269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.104244 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-h59hx"] Oct 30 19:33:55.105549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.105528 6907 scope.go:122] "RemoveContainer" containerID="3d6c7096815444383927ac19bb876def50dd9eb9599a7db6a93a8d6e61bae26a" Oct 30 19:33:55.118582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.118524 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:33:55.123793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.123767 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-g7q2t"] Oct 30 19:33:55.131924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.131894 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:33:55.138266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.138204 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-w4tsk"] Oct 30 19:33:55.150839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.150812 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:33:55.156038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.156010 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3bc085-openshift-default-77b977b9c5-xd5nc"] Oct 30 19:33:55.241137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.241113 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:33:55.638225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:55.638189 6907 scope.go:122] "RemoveContainer" containerID="c4d16586d8d2de64cc6fd0d230553c413151236663722db28a045d8e3d21848d" Oct 30 19:33:56.085261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.085215 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-n2bt8" event={"ID":"e9184944-65a5-4f9a-812e-d5d29a743897","Type":"ContainerStarted","Data":"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068"} Oct 30 19:33:56.085742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.085412 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:33:56.109332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.109280 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78455b774c-n2bt8" podStartSLOduration=2.536431253 podStartE2EDuration="6.109266169s" totalImagesPullingTime="3.572834916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:56.107895302 +0000 UTC m=+842.084121260" watchObservedRunningTime="2026-10-30 19:33:56.109266169 +0000 UTC m=+842.085492104" Oct 30 19:33:56.149465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.149410 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" path="/var/lib/kubelet/pods/60a7879a-cccd-4d31-b5f2-d56cafd48616/volumes" Oct 30 19:33:56.149914 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.149899 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" path="/var/lib/kubelet/pods/a56c4323-7158-4170-9d3a-81256d394f1a/volumes" Oct 30 19:33:56.150295 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.150282 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" path="/var/lib/kubelet/pods/cf26f935-3edf-458a-a38a-7a0ce517d86b/volumes" Oct 30 19:33:56.151056 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:33:56.151041 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" path="/var/lib/kubelet/pods/e8330f71-b86b-46fe-bb16-9e7c979308a8/volumes" Oct 30 19:34:07.095310 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:07.095274 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:34:07.736364 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:34:07.736326 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:34:13.879470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.879412 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n"] Oct 30 19:34:13.880121 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880101 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880124 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880147 6907 cpu_manager.go:414] "containerMap: removing container" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880156 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880186 6907 cpu_manager.go:414] "containerMap: removing container" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880193 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880211 6907 cpu_manager.go:414] "containerMap: removing container" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.880227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880218 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880244 6907 cpu_manager.go:414] "containerMap: removing container" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880252 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880365 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56c4323-7158-4170-9d3a-81256d394f1a" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880382 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8330f71-b86b-46fe-bb16-9e7c979308a8" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880393 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a7879a-cccd-4d31-b5f2-d56cafd48616" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880404 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.880614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.880415 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf26f935-3edf-458a-a38a-7a0ce517d86b" containerName="istio-proxy" Oct 30 19:34:13.884067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.884044 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:13.888239 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.888213 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:34:13.888385 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.888295 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:34:13.891685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.891657 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n"] Oct 30 19:34:13.914830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.914793 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:34:13.915017 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.914920 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:34:13.991305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.991274 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:13.991507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.991325 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:13.991507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:13.991487 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.092063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.092025 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.092246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.092103 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.092246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.092138 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.092831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.092806 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.094411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.094389 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.102291 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.102266 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\") pod \"maas-api-revoke-sub-85237e45-4z25n\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.201122 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.201036 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:14.487478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:14.484968 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n"] Oct 30 19:34:14.487478 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:34:14.485849 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc27de4a_58b6_4771_a4e9_d119fd177944.slice/crio-f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67 WatchSource:0}: Error finding container f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67: Status 404 returned error can't find the container with id f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67 Oct 30 19:34:15.218195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:15.218083 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" event={"ID":"fc27de4a-58b6-4771-a4e9-d119fd177944","Type":"ContainerStarted","Data":"64f58ee84c898e29b22aaf12deb2f71f08d5ab167b1d1bd1b691689782bacdaa"} Oct 30 19:34:15.218195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:15.218153 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" event={"ID":"fc27de4a-58b6-4771-a4e9-d119fd177944","Type":"ContainerStarted","Data":"f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67"} Oct 30 19:34:15.237925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:15.237874 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" podStartSLOduration=2.237862059 podStartE2EDuration="2.237862059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:15.237672905 +0000 UTC m=+861.213898849" watchObservedRunningTime="2026-10-30 19:34:15.237862059 +0000 UTC m=+861.214088003" Oct 30 19:34:16.226832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:16.226796 6907 generic.go:356] "Generic (PLEG): container finished" podID="fc27de4a-58b6-4771-a4e9-d119fd177944" containerID="64f58ee84c898e29b22aaf12deb2f71f08d5ab167b1d1bd1b691689782bacdaa" exitCode=0 Oct 30 19:34:16.227232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:16.226887 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" event={"ID":"fc27de4a-58b6-4771-a4e9-d119fd177944","Type":"ContainerDied","Data":"64f58ee84c898e29b22aaf12deb2f71f08d5ab167b1d1bd1b691689782bacdaa"} Oct 30 19:34:17.383463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.383416 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:17.527793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.527706 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\") pod \"fc27de4a-58b6-4771-a4e9-d119fd177944\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " Oct 30 19:34:17.527942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.527835 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\") pod \"fc27de4a-58b6-4771-a4e9-d119fd177944\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " Oct 30 19:34:17.527942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.527884 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\") pod \"fc27de4a-58b6-4771-a4e9-d119fd177944\" (UID: \"fc27de4a-58b6-4771-a4e9-d119fd177944\") " Oct 30 19:34:17.528147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.528116 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca" pod "fc27de4a-58b6-4771-a4e9-d119fd177944" (UID: "fc27de4a-58b6-4771-a4e9-d119fd177944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:34:17.529955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.529934 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7" pod "fc27de4a-58b6-4771-a4e9-d119fd177944" (UID: "fc27de4a-58b6-4771-a4e9-d119fd177944"). InnerVolumeSpecName "kube-api-access-tz7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:17.530030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.529934 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca" pod "fc27de4a-58b6-4771-a4e9-d119fd177944" (UID: "fc27de4a-58b6-4771-a4e9-d119fd177944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:17.629263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.629222 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:17.629263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.629259 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/fc27de4a-58b6-4771-a4e9-d119fd177944-kube-api-access-tz7p7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:17.629263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:17.629271 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc27de4a-58b6-4771-a4e9-d119fd177944-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:18.243027 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:18.242989 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" event={"ID":"fc27de4a-58b6-4771-a4e9-d119fd177944","Type":"ContainerDied","Data":"f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67"} Oct 30 19:34:18.243210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:18.243051 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f47d9274cff5bebbb7418bf107ab4bb67c45b03baf5dcd3712ceaa17130a5c67" Oct 30 19:34:18.243210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:18.242996 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n" Oct 30 19:34:25.590436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:25.590398 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:34:25.590930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:25.590656 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78455b774c-n2bt8" podUID="e9184944-65a5-4f9a-812e-d5d29a743897" containerName="manager" containerID="cri-o://20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068" gracePeriod=30 Oct 30 19:34:25.834012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:25.833984 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:34:26.018409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.018369 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\") pod \"e9184944-65a5-4f9a-812e-d5d29a743897\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " Oct 30 19:34:26.018614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.018490 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\" (UniqueName: \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\") pod \"e9184944-65a5-4f9a-812e-d5d29a743897\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " Oct 30 19:34:26.018614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.018560 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\") pod \"e9184944-65a5-4f9a-812e-d5d29a743897\" (UID: \"e9184944-65a5-4f9a-812e-d5d29a743897\") " Oct 30 19:34:26.020659 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.020623 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk" pod "e9184944-65a5-4f9a-812e-d5d29a743897" (UID: "e9184944-65a5-4f9a-812e-d5d29a743897"). InnerVolumeSpecName "kube-api-access-qmnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:26.020789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.020694 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert" pod "e9184944-65a5-4f9a-812e-d5d29a743897" (UID: "e9184944-65a5-4f9a-812e-d5d29a743897"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:26.020789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.020706 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs" pod "e9184944-65a5-4f9a-812e-d5d29a743897" (UID: "e9184944-65a5-4f9a-812e-d5d29a743897"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:34:26.119850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.119797 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.119850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.119845 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnsk\" (UniqueName: \"kubernetes.io/projected/e9184944-65a5-4f9a-812e-d5d29a743897-kube-api-access-qmnsk\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.119850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.119858 6907 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9184944-65a5-4f9a-812e-d5d29a743897-webhook-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:26.298815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.298726 6907 generic.go:356] "Generic (PLEG): container finished" podID="e9184944-65a5-4f9a-812e-d5d29a743897" containerID="20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068" exitCode=0 Oct 30 19:34:26.298815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.298787 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-n2bt8" Oct 30 19:34:26.299047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.298808 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-n2bt8" event={"ID":"e9184944-65a5-4f9a-812e-d5d29a743897","Type":"ContainerDied","Data":"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068"} Oct 30 19:34:26.299047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.298865 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-n2bt8" event={"ID":"e9184944-65a5-4f9a-812e-d5d29a743897","Type":"ContainerDied","Data":"28eabec80fdbdff51193103168798e763a00c0e2e0513494d33c48fa39fa55b8"} Oct 30 19:34:26.299047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.298887 6907 scope.go:122] "RemoveContainer" containerID="20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068" Oct 30 19:34:26.309138 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.309119 6907 scope.go:122] "RemoveContainer" containerID="20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068" Oct 30 19:34:26.309416 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:34:26.309395 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068\": container with ID starting with 20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068 not found: ID does not exist" containerID="20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068" Oct 30 19:34:26.309505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.309425 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068"} err="failed to get container status \"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068\": rpc error: code = NotFound desc = could not find container \"20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068\": container with ID starting with 20b2dad47a02d3774ac449e04245cf14e72eb70f017aa1e7b2788381e8309068 not found: ID does not exist" Oct 30 19:34:26.318664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.318640 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:34:26.323169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:26.323148 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78455b774c-n2bt8"] Oct 30 19:34:27.543771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.543734 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78455b774c-7466n"] Oct 30 19:34:27.544208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544194 6907 cpu_manager.go:414] "containerMap: removing container" podUID="e9184944-65a5-4f9a-812e-d5d29a743897" containerName="manager" Oct 30 19:34:27.544250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544210 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9184944-65a5-4f9a-812e-d5d29a743897" containerName="manager" Oct 30 19:34:27.544281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544252 6907 cpu_manager.go:414] "containerMap: removing container" podUID="fc27de4a-58b6-4771-a4e9-d119fd177944" containerName="revoke-keys" Oct 30 19:34:27.544281 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544258 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc27de4a-58b6-4771-a4e9-d119fd177944" containerName="revoke-keys" Oct 30 19:34:27.544348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544324 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc27de4a-58b6-4771-a4e9-d119fd177944" containerName="revoke-keys" Oct 30 19:34:27.544348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.544334 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9184944-65a5-4f9a-812e-d5d29a743897" containerName="manager" Oct 30 19:34:27.550168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.550148 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.554598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.554573 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:34:27.557272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.557247 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l2kjc\"" Oct 30 19:34:27.557407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.557350 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:34:27.557407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.557369 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:34:27.557615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.557582 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78455b774c-7466n"] Oct 30 19:34:27.738609 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.738579 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-webhook-cert\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.738780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.738724 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcgt\" (UniqueName: \"kubernetes.io/projected/d618490d-5e22-4274-833c-ab8d9b95453e-kube-api-access-6wcgt\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.738780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.738776 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-metrics-certs\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.839803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.839706 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcgt\" (UniqueName: \"kubernetes.io/projected/d618490d-5e22-4274-833c-ab8d9b95453e-kube-api-access-6wcgt\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.839803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.839774 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-metrics-certs\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.840030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.839829 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-webhook-cert\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.842262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.842229 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-webhook-cert\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.842262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.842252 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d618490d-5e22-4274-833c-ab8d9b95453e-metrics-certs\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.848841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.848813 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcgt\" (UniqueName: \"kubernetes.io/projected/d618490d-5e22-4274-833c-ab8d9b95453e-kube-api-access-6wcgt\") pod \"maas-controller-78455b774c-7466n\" (UID: \"d618490d-5e22-4274-833c-ab8d9b95453e\") " pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:27.866702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:27.866670 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:28.148459 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:34:28.148414 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd618490d_5e22_4274_833c_ab8d9b95453e.slice/crio-199e3d7cfd8ea50fdb063b117060ba975ccfd2da5c4b958e3689941a097266be WatchSource:0}: Error finding container 199e3d7cfd8ea50fdb063b117060ba975ccfd2da5c4b958e3689941a097266be: Status 404 returned error can't find the container with id 199e3d7cfd8ea50fdb063b117060ba975ccfd2da5c4b958e3689941a097266be Oct 30 19:34:28.149800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:28.149772 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9184944-65a5-4f9a-812e-d5d29a743897" path="/var/lib/kubelet/pods/e9184944-65a5-4f9a-812e-d5d29a743897/volumes" Oct 30 19:34:28.150089 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:28.150074 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78455b774c-7466n"] Oct 30 19:34:28.314459 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:28.314414 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-7466n" event={"ID":"d618490d-5e22-4274-833c-ab8d9b95453e","Type":"ContainerStarted","Data":"199e3d7cfd8ea50fdb063b117060ba975ccfd2da5c4b958e3689941a097266be"} Oct 30 19:34:29.324272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:29.324232 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78455b774c-7466n" event={"ID":"d618490d-5e22-4274-833c-ab8d9b95453e","Type":"ContainerStarted","Data":"a991fe91b1a9fa7b24f6ef30fe9d602bb438cfdf3ebe1510925852cea897d10e"} Oct 30 19:34:29.324682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:29.324325 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:29.347978 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:29.347912 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78455b774c-7466n" podStartSLOduration=1.7386764289999999 podStartE2EDuration="2.347896984s" totalImagesPullingTime="609.220555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:29.343859031 +0000 UTC m=+875.320084974" watchObservedRunningTime="2026-10-30 19:34:29.347896984 +0000 UTC m=+875.324122929" Oct 30 19:34:31.034859 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:31.034826 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v"] Oct 30 19:34:31.039751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:31.039723 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317ba8f4-gxf5v"] Oct 30 19:34:32.037773 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:32.037739 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv"] Oct 30 19:34:32.040656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:32.040630 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebd801da-zktlv"] Oct 30 19:34:32.150904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:32.150870 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147168d7-4250-4442-9d13-b15fbfd2c936" path="/var/lib/kubelet/pods/147168d7-4250-4442-9d13-b15fbfd2c936/volumes" Oct 30 19:34:32.151296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:32.151282 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ce8590-3148-4101-99f7-7bf3d0f1ac15" path="/var/lib/kubelet/pods/36ce8590-3148-4101-99f7-7bf3d0f1ac15/volumes" Oct 30 19:34:33.028848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:33.028813 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6"] Oct 30 19:34:33.031996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:33.031973 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fe70ee8-f4gp6"] Oct 30 19:34:34.150745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:34.150701 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fb3dd2-2329-47a8-b148-e3d10d155e47" path="/var/lib/kubelet/pods/a5fb3dd2-2329-47a8-b148-e3d10d155e47/volumes" Oct 30 19:34:37.736178 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:34:37.736137 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:34:40.334067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:40.334035 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78455b774c-7466n" Oct 30 19:34:42.025624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:42.025589 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9"] Oct 30 19:34:42.028787 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:42.028766 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8060255e-rttj9"] Oct 30 19:34:42.150135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:42.150097 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c596a20-cd10-4925-bcc0-1fc3089bf53d" path="/var/lib/kubelet/pods/5c596a20-cd10-4925-bcc0-1fc3089bf53d/volumes" Oct 30 19:34:44.036031 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:44.035994 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz"] Oct 30 19:34:44.040429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:44.040402 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-023c8182-g97pz"] Oct 30 19:34:44.155317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:44.155271 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f359cbf3-3422-479d-a9a8-a7a95a17ea96" path="/var/lib/kubelet/pods/f359cbf3-3422-479d-a9a8-a7a95a17ea96/volumes" Oct 30 19:34:51.029195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:51.029159 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44"] Oct 30 19:34:51.031967 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:51.031946 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-880b704d-m7s44"] Oct 30 19:34:52.150630 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:52.150598 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177391a3-ec28-405c-817f-95aba15c9bde" path="/var/lib/kubelet/pods/177391a3-ec28-405c-817f-95aba15c9bde/volumes" Oct 30 19:34:54.291151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:54.291121 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:34:54.296463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:54.296426 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:34:54.309926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:54.309902 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:34:54.314968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:54.314949 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:34:57.024202 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:57.024164 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq"] Oct 30 19:34:57.027079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:57.027054 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6bc7d0c-vn4pq"] Oct 30 19:34:58.028289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:58.028241 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8"] Oct 30 19:34:58.031548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:58.031513 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1225c727-sbvc8"] Oct 30 19:34:58.149929 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:58.149892 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a3049e-e477-40e2-9c10-99d258d6ae99" path="/var/lib/kubelet/pods/37a3049e-e477-40e2-9c10-99d258d6ae99/volumes" Oct 30 19:34:58.150361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:34:58.150346 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721819bb-d6d8-4357-9fbd-e868d8343dd5" path="/var/lib/kubelet/pods/721819bb-d6d8-4357-9fbd-e868d8343dd5/volumes" Oct 30 19:35:02.810157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:02.810118 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:35:07.024289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.024182 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn"] Oct 30 19:35:07.027132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.027101 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e916cf6-9wqtn"] Oct 30 19:35:07.612564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.612531 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr"] Oct 30 19:35:07.618019 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.617999 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.622360 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.622317 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:07.622524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.622399 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:07.624697 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.624672 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr"] Oct 30 19:35:07.701926 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.701893 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:07.702128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.702053 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:35:07.738079 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:35:07.738042 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:35:07.738694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.738670 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.738813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.738705 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.738813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.738734 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b2n\" (UniqueName: \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.839921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.839881 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.840109 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.839931 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.840109 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.839970 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b2n\" (UniqueName: \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.840673 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.840650 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.842317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.842296 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.852075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.852050 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b2n\" (UniqueName: \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\") pod \"maas-api-revoke-sub-c1e271a9-8knxr\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:07.935352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:07.935269 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:08.151492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:08.151436 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30917b9a-0bc0-4727-b757-812f64ff420b" path="/var/lib/kubelet/pods/30917b9a-0bc0-4727-b757-812f64ff420b/volumes" Oct 30 19:35:08.214140 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:08.214107 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr"] Oct 30 19:35:08.215457 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:35:08.215420 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f19bfa_c8fb_44f3_865c_7b303bde8739.slice/crio-1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d WatchSource:0}: Error finding container 1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d: Status 404 returned error can't find the container with id 1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d Oct 30 19:35:08.640181 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:08.640142 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" event={"ID":"49f19bfa-c8fb-44f3-865c-7b303bde8739","Type":"ContainerStarted","Data":"00ed70686c115dd17925237446efed6ce6a05783cf9010e1a872881eb54d7106"} Oct 30 19:35:08.640390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:08.640195 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" event={"ID":"49f19bfa-c8fb-44f3-865c-7b303bde8739","Type":"ContainerStarted","Data":"1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d"} Oct 30 19:35:08.659358 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:08.659302 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" podStartSLOduration=1.6592866499999999 podStartE2EDuration="1.65928665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:08.657522442 +0000 UTC m=+914.633748385" watchObservedRunningTime="2026-10-30 19:35:08.65928665 +0000 UTC m=+914.635512672" Oct 30 19:35:09.649076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:09.649041 6907 generic.go:356] "Generic (PLEG): container finished" podID="49f19bfa-c8fb-44f3-865c-7b303bde8739" containerID="00ed70686c115dd17925237446efed6ce6a05783cf9010e1a872881eb54d7106" exitCode=0 Oct 30 19:35:09.649498 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:09.649132 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" event={"ID":"49f19bfa-c8fb-44f3-865c-7b303bde8739","Type":"ContainerDied","Data":"00ed70686c115dd17925237446efed6ce6a05783cf9010e1a872881eb54d7106"} Oct 30 19:35:10.797332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.797307 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:10.871219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.871170 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\") pod \"49f19bfa-c8fb-44f3-865c-7b303bde8739\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " Oct 30 19:35:10.871425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.871227 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\" (UniqueName: \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\") pod \"49f19bfa-c8fb-44f3-865c-7b303bde8739\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " Oct 30 19:35:10.871425 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.871389 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\") pod \"49f19bfa-c8fb-44f3-865c-7b303bde8739\" (UID: \"49f19bfa-c8fb-44f3-865c-7b303bde8739\") " Oct 30 19:35:10.871762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.871730 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca" pod "49f19bfa-c8fb-44f3-865c-7b303bde8739" (UID: "49f19bfa-c8fb-44f3-865c-7b303bde8739"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:10.873410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.873385 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca" pod "49f19bfa-c8fb-44f3-865c-7b303bde8739" (UID: "49f19bfa-c8fb-44f3-865c-7b303bde8739"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:10.873523 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.873432 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n" pod "49f19bfa-c8fb-44f3-865c-7b303bde8739" (UID: "49f19bfa-c8fb-44f3-865c-7b303bde8739"). InnerVolumeSpecName "kube-api-access-v2b2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:10.972313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.972281 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:10.972313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.972314 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49f19bfa-c8fb-44f3-865c-7b303bde8739-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:10.972595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:10.972324 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2b2n\" (UniqueName: \"kubernetes.io/projected/49f19bfa-c8fb-44f3-865c-7b303bde8739-kube-api-access-v2b2n\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:11.034495 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:11.034419 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px"] Oct 30 19:35:11.039271 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:11.039241 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b41ff9f-rh2px"] Oct 30 19:35:11.663795 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:11.663759 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" Oct 30 19:35:11.664084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:11.663791 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr" event={"ID":"49f19bfa-c8fb-44f3-865c-7b303bde8739","Type":"ContainerDied","Data":"1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d"} Oct 30 19:35:11.664084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:11.663850 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1525c004c77fc81c77a769eccc038ba5033d9232bb96f4b64ba5785693f31e7d" Oct 30 19:35:12.151049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:12.151017 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4cdf11-f521-403d-9465-69c59fcb1418" path="/var/lib/kubelet/pods/7d4cdf11-f521-403d-9465-69c59fcb1418/volumes" Oct 30 19:35:18.518057 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:18.518020 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:35:22.040217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:22.040182 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw"] Oct 30 19:35:22.056736 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:22.056664 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca035442-t78lw"] Oct 30 19:35:22.171470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:22.162465 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d00d1c-8aa9-4a5d-b4ac-e7eda727f981" path="/var/lib/kubelet/pods/83d00d1c-8aa9-4a5d-b4ac-e7eda727f981/volumes" Oct 30 19:35:22.370656 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:22.370561 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:35:26.341155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.341112 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm"] Oct 30 19:35:26.341821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.341800 6907 cpu_manager.go:414] "containerMap: removing container" podUID="49f19bfa-c8fb-44f3-865c-7b303bde8739" containerName="revoke-keys" Oct 30 19:35:26.341917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.341823 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f19bfa-c8fb-44f3-865c-7b303bde8739" containerName="revoke-keys" Oct 30 19:35:26.341978 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.341951 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f19bfa-c8fb-44f3-865c-7b303bde8739" containerName="revoke-keys" Oct 30 19:35:26.345647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.345625 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.350775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.350742 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:26.350911 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.350829 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:26.350979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.350942 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:35:26.351106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.351080 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:26.355412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.355385 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm"] Oct 30 19:35:26.483620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.483579 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.483620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.483624 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.483852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.483653 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qk99\" (UniqueName: \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.584820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.584781 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.584820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.584826 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.585037 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.584866 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qk99\" (UniqueName: \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.585489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.585434 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.587250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.587228 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.595946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.595873 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qk99\" (UniqueName: \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\") pod \"maas-api-revoke-sub-2e1f5b5f-jwqbm\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.662315 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.662272 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:26.942895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:26.942864 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm"] Oct 30 19:35:26.943747 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:35:26.943717 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef09ce1_64fc_402f_a531_3794c7c201d6.slice/crio-95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6 WatchSource:0}: Error finding container 95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6: Status 404 returned error can't find the container with id 95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6 Oct 30 19:35:27.815190 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:27.815153 6907 generic.go:356] "Generic (PLEG): container finished" podID="0ef09ce1-64fc-402f-a531-3794c7c201d6" containerID="d1a487da3485b23533b28cca0a3092a0a3260ba136794cee74c7a0a6c5c5d70c" exitCode=0 Oct 30 19:35:27.815683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:27.815241 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" event={"ID":"0ef09ce1-64fc-402f-a531-3794c7c201d6","Type":"ContainerDied","Data":"d1a487da3485b23533b28cca0a3092a0a3260ba136794cee74c7a0a6c5c5d70c"} Oct 30 19:35:27.815683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:27.815289 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" event={"ID":"0ef09ce1-64fc-402f-a531-3794c7c201d6","Type":"ContainerStarted","Data":"95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6"} Oct 30 19:35:28.963437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:28.963407 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:29.109321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.109234 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\") pod \"0ef09ce1-64fc-402f-a531-3794c7c201d6\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " Oct 30 19:35:29.109539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.109364 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\") pod \"0ef09ce1-64fc-402f-a531-3794c7c201d6\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " Oct 30 19:35:29.109539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.109418 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\" (UniqueName: \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\") pod \"0ef09ce1-64fc-402f-a531-3794c7c201d6\" (UID: \"0ef09ce1-64fc-402f-a531-3794c7c201d6\") " Oct 30 19:35:29.109768 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.109743 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca" pod "0ef09ce1-64fc-402f-a531-3794c7c201d6" (UID: "0ef09ce1-64fc-402f-a531-3794c7c201d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:29.111323 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.111294 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca" pod "0ef09ce1-64fc-402f-a531-3794c7c201d6" (UID: "0ef09ce1-64fc-402f-a531-3794c7c201d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:29.111533 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.111375 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99" pod "0ef09ce1-64fc-402f-a531-3794c7c201d6" (UID: "0ef09ce1-64fc-402f-a531-3794c7c201d6"). InnerVolumeSpecName "kube-api-access-6qk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:29.210748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.210712 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.210748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.210744 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qk99\" (UniqueName: \"kubernetes.io/projected/0ef09ce1-64fc-402f-a531-3794c7c201d6-kube-api-access-6qk99\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.210748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.210757 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ef09ce1-64fc-402f-a531-3794c7c201d6-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:29.831139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.831096 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" event={"ID":"0ef09ce1-64fc-402f-a531-3794c7c201d6","Type":"ContainerDied","Data":"95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6"} Oct 30 19:35:29.831139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.831125 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm" Oct 30 19:35:29.831362 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:29.831158 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95926c05e888dd5782536dce5ae94a4c5716eefde954e0edb9350489a6e23aa6" Oct 30 19:35:33.027041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:33.026985 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl"] Oct 30 19:35:33.030305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:33.030281 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2464616-lb7tl"] Oct 30 19:35:34.157196 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:34.157157 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f045a0-31c7-4737-879b-44b21b296caa" path="/var/lib/kubelet/pods/a1f045a0-31c7-4737-879b-44b21b296caa/volumes" Oct 30 19:35:37.266454 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.266416 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:35:37.364105 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.364065 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm"] Oct 30 19:35:37.364573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.364560 6907 cpu_manager.go:414] "containerMap: removing container" podUID="0ef09ce1-64fc-402f-a531-3794c7c201d6" containerName="revoke-keys" Oct 30 19:35:37.364573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.364574 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef09ce1-64fc-402f-a531-3794c7c201d6" containerName="revoke-keys" Oct 30 19:35:37.364711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.364670 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef09ce1-64fc-402f-a531-3794c7c201d6" containerName="revoke-keys" Oct 30 19:35:37.368285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.368263 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.375106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.375049 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:35:37.375106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.375052 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:35:37.375106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.375084 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:35:37.375106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.375090 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:35:37.388094 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.388058 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm"] Oct 30 19:35:37.500212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.500172 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.500422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.500317 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.500422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.500349 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgktk\" (UniqueName: \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.605108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.604540 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.605321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.605297 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgktk\" (UniqueName: \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.606006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.605428 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.606237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.606209 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.609009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.608982 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.615412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.615361 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgktk\" (UniqueName: \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\") pod \"maas-api-revoke-sub-ec843018-45vnm\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.682580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.682514 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:37.744235 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:35:37.744175 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:35:37.978585 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:37.978551 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm"] Oct 30 19:35:37.980579 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:35:37.980553 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e77756_b290_461c_82b1_0ca8570eb22a.slice/crio-b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238 WatchSource:0}: Error finding container b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238: Status 404 returned error can't find the container with id b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238 Oct 30 19:35:38.893625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:38.893588 6907 generic.go:356] "Generic (PLEG): container finished" podID="52e77756-b290-461c-82b1-0ca8570eb22a" containerID="6ae099040ee7b433e25e49a42edcc1b31a046d70fee12256a120b464c4419372" exitCode=0 Oct 30 19:35:38.894022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:38.893639 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" event={"ID":"52e77756-b290-461c-82b1-0ca8570eb22a","Type":"ContainerDied","Data":"6ae099040ee7b433e25e49a42edcc1b31a046d70fee12256a120b464c4419372"} Oct 30 19:35:38.894022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:38.893694 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" event={"ID":"52e77756-b290-461c-82b1-0ca8570eb22a","Type":"ContainerStarted","Data":"b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238"} Oct 30 19:35:40.103741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.103705 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:40.253617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.253561 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\" (UniqueName: \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\") pod \"52e77756-b290-461c-82b1-0ca8570eb22a\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " Oct 30 19:35:40.253850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.253682 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\") pod \"52e77756-b290-461c-82b1-0ca8570eb22a\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " Oct 30 19:35:40.253850 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.253784 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\") pod \"52e77756-b290-461c-82b1-0ca8570eb22a\" (UID: \"52e77756-b290-461c-82b1-0ca8570eb22a\") " Oct 30 19:35:40.254573 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.254540 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca" pod "52e77756-b290-461c-82b1-0ca8570eb22a" (UID: "52e77756-b290-461c-82b1-0ca8570eb22a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:35:40.257506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.257463 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca" pod "52e77756-b290-461c-82b1-0ca8570eb22a" (UID: "52e77756-b290-461c-82b1-0ca8570eb22a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:35:40.283420 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.283354 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk" pod "52e77756-b290-461c-82b1-0ca8570eb22a" (UID: "52e77756-b290-461c-82b1-0ca8570eb22a"). InnerVolumeSpecName "kube-api-access-vgktk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:40.362467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.358540 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:40.362467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.358595 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e77756-b290-461c-82b1-0ca8570eb22a-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:40.362467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.358617 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgktk\" (UniqueName: \"kubernetes.io/projected/52e77756-b290-461c-82b1-0ca8570eb22a-kube-api-access-vgktk\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:40.924470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.922877 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" event={"ID":"52e77756-b290-461c-82b1-0ca8570eb22a","Type":"ContainerDied","Data":"b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238"} Oct 30 19:35:40.924470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.922955 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9862f2ef00c5a990025f973e3531fea15a35f4b2d703ac4bb362ee492dda238" Oct 30 19:35:40.924470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:40.923027 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm" Oct 30 19:35:50.534699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:50.534662 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:35:55.824401 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.824367 6907 scope.go:122] "RemoveContainer" containerID="f49c970ef02f73f2dc552098bae8dce9b799e4a16e24c02f97446a94d141fc95" Oct 30 19:35:55.834238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.834216 6907 scope.go:122] "RemoveContainer" containerID="c689c82ebeab78eff1cdbbfc44c7225b52d2e7661996da3d6b82c8c341179d10" Oct 30 19:35:55.843489 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.843468 6907 scope.go:122] "RemoveContainer" containerID="f5c928246e8c4a723706e6e525ad8e2d50c64c87e7cb823a4123d276be670587" Oct 30 19:35:55.852043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.852020 6907 scope.go:122] "RemoveContainer" containerID="11e184fae3a05457f9c60f383476680ac4dde7602153a4e521b3ba28f030a0d5" Oct 30 19:35:55.860791 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.860774 6907 scope.go:122] "RemoveContainer" containerID="2dad88cc8c0d001f917240442ab8bea7cff7e7057f49c70e09b93d2d604d851e" Oct 30 19:35:55.869414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.869391 6907 scope.go:122] "RemoveContainer" containerID="c88ba2fdab82033f0b14d2e763bccc6ac5770f38b6913374d7d3c39b8a1ad3de" Oct 30 19:35:55.879432 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.879406 6907 scope.go:122] "RemoveContainer" containerID="39237f8a9a608e6694261f67adb9deb2aa0248ce101932f80e0713fba5aa2db4" Oct 30 19:35:55.888830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:55.888805 6907 scope.go:122] "RemoveContainer" containerID="e417c72171953aecaec01d7be5774a78e3bf0227ef30a7fc3e4fd399ee038820" Oct 30 19:35:57.026122 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:57.026085 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc"] Oct 30 19:35:57.029402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:57.029370 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1dc9336-nh2hc"] Oct 30 19:35:58.149686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:35:58.149649 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2902e97-612a-491d-aab8-ea1c6cf83881" path="/var/lib/kubelet/pods/a2902e97-612a-491d-aab8-ea1c6cf83881/volumes" Oct 30 19:36:01.083276 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.083242 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg"] Oct 30 19:36:01.083765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.083751 6907 cpu_manager.go:414] "containerMap: removing container" podUID="52e77756-b290-461c-82b1-0ca8570eb22a" containerName="revoke-keys" Oct 30 19:36:01.083820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.083767 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e77756-b290-461c-82b1-0ca8570eb22a" containerName="revoke-keys" Oct 30 19:36:01.083942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.083929 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e77756-b290-461c-82b1-0ca8570eb22a" containerName="revoke-keys" Oct 30 19:36:01.089101 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.089080 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.093683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.093643 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:01.093842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.093721 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:01.093921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.093857 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:01.094563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.094541 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:36:01.096036 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.096017 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg"] Oct 30 19:36:01.276615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.276581 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.276843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.276639 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.276843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.276663 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzwx\" (UniqueName: \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.378109 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.378004 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.378109 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.378064 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzwx\" (UniqueName: \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.378314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.378165 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.378743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.378709 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.380516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.380494 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.386466 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.386414 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzwx\" (UniqueName: \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\") pod \"maas-api-revoke-sub-046e1343-vnfvg\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.405408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.405387 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:01.683509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:01.683476 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg"] Oct 30 19:36:01.684475 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:36:01.684421 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c02eab_19ea_4646_b0a4_2f6b47240a76.slice/crio-bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7 WatchSource:0}: Error finding container bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7: Status 404 returned error can't find the container with id bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7 Oct 30 19:36:02.141105 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:02.141069 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" event={"ID":"d1c02eab-19ea-4646-b0a4-2f6b47240a76","Type":"ContainerStarted","Data":"e92a0e7aaa9c16aadde5effe2e36d1e47348da60e93fd9051ec023142cbc118f"} Oct 30 19:36:02.141536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:02.141125 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" event={"ID":"d1c02eab-19ea-4646-b0a4-2f6b47240a76","Type":"ContainerStarted","Data":"bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7"} Oct 30 19:36:02.158935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:02.158886 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" podStartSLOduration=1.158870555 podStartE2EDuration="1.158870555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:02.157792474 +0000 UTC m=+968.134018419" watchObservedRunningTime="2026-10-30 19:36:02.158870555 +0000 UTC m=+968.135096499" Oct 30 19:36:03.149290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:03.149254 6907 generic.go:356] "Generic (PLEG): container finished" podID="d1c02eab-19ea-4646-b0a4-2f6b47240a76" containerID="e92a0e7aaa9c16aadde5effe2e36d1e47348da60e93fd9051ec023142cbc118f" exitCode=0 Oct 30 19:36:03.149743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:03.149340 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" event={"ID":"d1c02eab-19ea-4646-b0a4-2f6b47240a76","Type":"ContainerDied","Data":"e92a0e7aaa9c16aadde5effe2e36d1e47348da60e93fd9051ec023142cbc118f"} Oct 30 19:36:04.301408 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.301380 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:04.409371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.409285 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\" (UniqueName: \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\") pod \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " Oct 30 19:36:04.409583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.409380 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\") pod \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " Oct 30 19:36:04.409583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.409475 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\") pod \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\" (UID: \"d1c02eab-19ea-4646-b0a4-2f6b47240a76\") " Oct 30 19:36:04.409812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.409791 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca" pod "d1c02eab-19ea-4646-b0a4-2f6b47240a76" (UID: "d1c02eab-19ea-4646-b0a4-2f6b47240a76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:04.411382 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.411356 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca" pod "d1c02eab-19ea-4646-b0a4-2f6b47240a76" (UID: "d1c02eab-19ea-4646-b0a4-2f6b47240a76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:04.411382 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.411356 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx" pod "d1c02eab-19ea-4646-b0a4-2f6b47240a76" (UID: "d1c02eab-19ea-4646-b0a4-2f6b47240a76"). InnerVolumeSpecName "kube-api-access-sqzwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:04.510924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.510885 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzwx\" (UniqueName: \"kubernetes.io/projected/d1c02eab-19ea-4646-b0a4-2f6b47240a76-kube-api-access-sqzwx\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:04.510924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.510919 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:04.511140 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:04.510931 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c02eab-19ea-4646-b0a4-2f6b47240a76-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:05.162761 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:05.162727 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" Oct 30 19:36:05.162947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:05.162759 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg" event={"ID":"d1c02eab-19ea-4646-b0a4-2f6b47240a76","Type":"ContainerDied","Data":"bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7"} Oct 30 19:36:05.162947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:05.162811 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdea26be20ded32d73bd76e5786b96d4e072dfacdae572a2795a0be10c0f00d7" Oct 30 19:36:07.736854 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:36:07.736781 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:36:11.033906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.033870 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9"] Oct 30 19:36:11.036823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.036792 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164203ac-6t9z9"] Oct 30 19:36:11.307512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.307403 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs"] Oct 30 19:36:11.307924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.307901 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d1c02eab-19ea-4646-b0a4-2f6b47240a76" containerName="revoke-keys" Oct 30 19:36:11.307965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.307925 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c02eab-19ea-4646-b0a4-2f6b47240a76" containerName="revoke-keys" Oct 30 19:36:11.308022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.308013 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c02eab-19ea-4646-b0a4-2f6b47240a76" containerName="revoke-keys" Oct 30 19:36:11.311243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.311227 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.316145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.316115 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:11.316313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.316242 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:11.316313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.316300 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:11.316430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.316318 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:36:11.319419 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.319388 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs"] Oct 30 19:36:11.378053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.378018 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.378240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.378098 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqs7\" (UniqueName: \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.378240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.378202 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.479791 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.479753 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.480427 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.479840 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqs7\" (UniqueName: \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.480724 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.480702 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.488405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.481119 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.488405 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.483833 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.491319 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.491290 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqs7\" (UniqueName: \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\") pod \"maas-api-revoke-sub-d3fba056-jrvcs\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.627995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.627896 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:11.906567 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.906530 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs"] Oct 30 19:36:11.908275 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:36:11.908242 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503f1bc1_ca11_4535_9a4e_cdb6cb6b1f25.slice/crio-a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460 WatchSource:0}: Error finding container a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460: Status 404 returned error can't find the container with id a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460 Oct 30 19:36:11.935203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:11.935174 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:36:12.149991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:12.149961 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca787e75-dcf6-4450-9725-406d89f8dd6f" path="/var/lib/kubelet/pods/ca787e75-dcf6-4450-9725-406d89f8dd6f/volumes" Oct 30 19:36:12.208142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:12.208055 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" event={"ID":"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25","Type":"ContainerStarted","Data":"a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460"} Oct 30 19:36:13.222568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:13.222022 6907 generic.go:356] "Generic (PLEG): container finished" podID="503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" containerID="a0391f7f32ed6d784a2c3e74fcff596c22314a7a2cf11c6f6ddf44f43860b7f6" exitCode=0 Oct 30 19:36:13.222568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:13.222132 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" event={"ID":"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25","Type":"ContainerDied","Data":"a0391f7f32ed6d784a2c3e74fcff596c22314a7a2cf11c6f6ddf44f43860b7f6"} Oct 30 19:36:15.002578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.002533 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:15.126640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.126592 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\") pod \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " Oct 30 19:36:15.126895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.126739 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\") pod \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " Oct 30 19:36:15.126895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.126813 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\" (UniqueName: \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\") pod \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\" (UID: \"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25\") " Oct 30 19:36:15.127079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.127050 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca" pod "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" (UID: "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:15.127298 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.127274 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:15.129115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.129089 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca" pod "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" (UID: "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:15.129115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.129107 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7" pod "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" (UID: "503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25"). InnerVolumeSpecName "kube-api-access-kpqs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:15.228309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.228275 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:15.228309 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.228305 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqs7\" (UniqueName: \"kubernetes.io/projected/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25-kube-api-access-kpqs7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:15.267595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.267563 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" event={"ID":"503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25","Type":"ContainerDied","Data":"a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460"} Oct 30 19:36:15.267744 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.267617 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32d3ae292b366a964735f01ec6d6c97faad95aaed32e8ac16d4b84d5f644460" Oct 30 19:36:15.267744 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:15.267567 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs" Oct 30 19:36:22.025621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:22.025580 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs"] Oct 30 19:36:22.029258 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:22.029233 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34e1be6f-7khrs"] Oct 30 19:36:22.124971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:22.124934 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:36:22.150782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:22.150749 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2228b87-c3bd-48df-a080-3fb7de832c68" path="/var/lib/kubelet/pods/f2228b87-c3bd-48df-a080-3fb7de832c68/volumes" Oct 30 19:36:27.427316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:27.426526 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:36:31.023413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:31.023369 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b"] Oct 30 19:36:31.025920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:31.025896 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dc79d9c-ntx9b"] Oct 30 19:36:32.057743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.057709 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk"] Oct 30 19:36:32.058171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.058158 6907 cpu_manager.go:414] "containerMap: removing container" podUID="503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" containerName="revoke-keys" Oct 30 19:36:32.058217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.058172 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" containerName="revoke-keys" Oct 30 19:36:32.058283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.058274 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" containerName="revoke-keys" Oct 30 19:36:32.062854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.062838 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.066737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.066711 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:32.066925 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.066722 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:32.067084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.067058 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:32.067208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.067128 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:36:32.070509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.070482 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk"] Oct 30 19:36:32.150050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.150017 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e7a0d9-0208-4efa-b2fc-d30df5899c46" path="/var/lib/kubelet/pods/46e7a0d9-0208-4efa-b2fc-d30df5899c46/volumes" Oct 30 19:36:32.173173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.173139 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.173370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.173188 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.173370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.173215 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mn2\" (UniqueName: \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.274583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.274546 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.274762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.274596 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.274762 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.274622 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mn2\" (UniqueName: \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.275234 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.275211 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.277149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.277116 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.283260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.283233 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mn2\" (UniqueName: \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\") pod \"maas-api-revoke-sub-ae594979-chvpk\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.385751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.385662 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:32.664807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:32.664778 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk"] Oct 30 19:36:33.440786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:33.440748 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" event={"ID":"3bc00f83-970a-4dda-acf8-3987a7784f94","Type":"ContainerStarted","Data":"580a833e78a57283de937c7846075c8ef214fe22d61f330b087581691df87650"} Oct 30 19:36:33.441171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:33.440803 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" event={"ID":"3bc00f83-970a-4dda-acf8-3987a7784f94","Type":"ContainerStarted","Data":"b09949e7d073ff840c252e307d6800a6482b6a9e3fe99fabefc5abe07361f4f2"} Oct 30 19:36:33.459171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:33.459114 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" podStartSLOduration=1.459099203 podStartE2EDuration="1.459099203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:33.457815761 +0000 UTC m=+999.434041709" watchObservedRunningTime="2026-10-30 19:36:33.459099203 +0000 UTC m=+999.435325224" Oct 30 19:36:34.448367 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:34.448330 6907 generic.go:356] "Generic (PLEG): container finished" podID="3bc00f83-970a-4dda-acf8-3987a7784f94" containerID="580a833e78a57283de937c7846075c8ef214fe22d61f330b087581691df87650" exitCode=0 Oct 30 19:36:34.448855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:34.448419 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" event={"ID":"3bc00f83-970a-4dda-acf8-3987a7784f94","Type":"ContainerDied","Data":"580a833e78a57283de937c7846075c8ef214fe22d61f330b087581691df87650"} Oct 30 19:36:35.593108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.593083 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:35.710524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.710400 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\") pod \"3bc00f83-970a-4dda-acf8-3987a7784f94\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " Oct 30 19:36:35.710524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.710516 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\") pod \"3bc00f83-970a-4dda-acf8-3987a7784f94\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " Oct 30 19:36:35.710781 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.710571 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\" (UniqueName: \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\") pod \"3bc00f83-970a-4dda-acf8-3987a7784f94\" (UID: \"3bc00f83-970a-4dda-acf8-3987a7784f94\") " Oct 30 19:36:35.710865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.710837 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca" pod "3bc00f83-970a-4dda-acf8-3987a7784f94" (UID: "3bc00f83-970a-4dda-acf8-3987a7784f94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:35.712723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.712690 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca" pod "3bc00f83-970a-4dda-acf8-3987a7784f94" (UID: "3bc00f83-970a-4dda-acf8-3987a7784f94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:35.712723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.712696 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2" pod "3bc00f83-970a-4dda-acf8-3987a7784f94" (UID: "3bc00f83-970a-4dda-acf8-3987a7784f94"). InnerVolumeSpecName "kube-api-access-h2mn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:35.811695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.811616 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:35.811695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.811649 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2mn2\" (UniqueName: \"kubernetes.io/projected/3bc00f83-970a-4dda-acf8-3987a7784f94-kube-api-access-h2mn2\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:35.811695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:35.811660 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc00f83-970a-4dda-acf8-3987a7784f94-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:36.462612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:36.462517 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" event={"ID":"3bc00f83-970a-4dda-acf8-3987a7784f94","Type":"ContainerDied","Data":"b09949e7d073ff840c252e307d6800a6482b6a9e3fe99fabefc5abe07361f4f2"} Oct 30 19:36:36.462612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:36.462538 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk" Oct 30 19:36:36.462612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:36.462575 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09949e7d073ff840c252e307d6800a6482b6a9e3fe99fabefc5abe07361f4f2" Oct 30 19:36:37.025200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:37.025157 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm"] Oct 30 19:36:37.028061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:37.028036 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962d6fe8-lp6gm"] Oct 30 19:36:37.735917 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:36:37.735868 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:36:38.149798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:38.149718 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48d1ec3-959e-4461-ab7e-d648a38119a8" path="/var/lib/kubelet/pods/e48d1ec3-959e-4461-ab7e-d648a38119a8/volumes" Oct 30 19:36:42.859969 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:42.859928 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:36:46.485786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:46.485745 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:36:49.025117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:49.025078 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w"] Oct 30 19:36:49.027599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:49.027573 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34097152-jxq2w"] Oct 30 19:36:50.149341 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.149296 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39133361-18df-4262-9e9b-93396062c1d3" path="/var/lib/kubelet/pods/39133361-18df-4262-9e9b-93396062c1d3/volumes" Oct 30 19:36:50.262475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.262422 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5"] Oct 30 19:36:50.262963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.262947 6907 cpu_manager.go:414] "containerMap: removing container" podUID="3bc00f83-970a-4dda-acf8-3987a7784f94" containerName="revoke-keys" Oct 30 19:36:50.263013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.262966 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc00f83-970a-4dda-acf8-3987a7784f94" containerName="revoke-keys" Oct 30 19:36:50.263072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.263063 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc00f83-970a-4dda-acf8-3987a7784f94" containerName="revoke-keys" Oct 30 19:36:50.267508 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.267489 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.271542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.271518 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:50.271542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.271544 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:50.271751 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.271575 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:36:50.271840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.271820 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:36:50.274835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.274809 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5"] Oct 30 19:36:50.303227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.303197 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.303416 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.303345 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.303540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.303522 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996hm\" (UniqueName: \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.404986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.404886 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.404986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.404971 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.405240 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.405047 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996hm\" (UniqueName: \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.405576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.405554 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.407373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.407353 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.414269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.414245 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-996hm\" (UniqueName: \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\") pod \"maas-api-revoke-sub-a4aef7d7-txrs5\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.582571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.582529 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:50.863807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:50.863777 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5"] Oct 30 19:36:50.866373 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:36:50.866343 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e294bea_7b40_452e_b272_92c9bea69bc4.slice/crio-ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75 WatchSource:0}: Error finding container ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75: Status 404 returned error can't find the container with id ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75 Oct 30 19:36:51.603164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:51.603116 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" event={"ID":"6e294bea-7b40-452e-b272-92c9bea69bc4","Type":"ContainerStarted","Data":"c7ac16082d26049caf37a476dd90cd4aebc128de677df8d4f7c7677ad9dbf0c8"} Oct 30 19:36:51.603651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:51.603183 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" event={"ID":"6e294bea-7b40-452e-b272-92c9bea69bc4","Type":"ContainerStarted","Data":"ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75"} Oct 30 19:36:51.623881 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:51.623831 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" podStartSLOduration=1.623816637 podStartE2EDuration="1.623816637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:51.621249562 +0000 UTC m=+1017.597475507" watchObservedRunningTime="2026-10-30 19:36:51.623816637 +0000 UTC m=+1017.600042580" Oct 30 19:36:52.619790 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:52.619752 6907 generic.go:356] "Generic (PLEG): container finished" podID="6e294bea-7b40-452e-b272-92c9bea69bc4" containerID="c7ac16082d26049caf37a476dd90cd4aebc128de677df8d4f7c7677ad9dbf0c8" exitCode=0 Oct 30 19:36:52.620230 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:52.619813 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" event={"ID":"6e294bea-7b40-452e-b272-92c9bea69bc4","Type":"ContainerDied","Data":"c7ac16082d26049caf37a476dd90cd4aebc128de677df8d4f7c7677ad9dbf0c8"} Oct 30 19:36:53.774339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.774312 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:53.838823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.838783 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\" (UniqueName: \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\") pod \"6e294bea-7b40-452e-b272-92c9bea69bc4\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " Oct 30 19:36:53.839018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.838838 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\") pod \"6e294bea-7b40-452e-b272-92c9bea69bc4\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " Oct 30 19:36:53.839018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.838922 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\") pod \"6e294bea-7b40-452e-b272-92c9bea69bc4\" (UID: \"6e294bea-7b40-452e-b272-92c9bea69bc4\") " Oct 30 19:36:53.839400 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.839369 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca" pod "6e294bea-7b40-452e-b272-92c9bea69bc4" (UID: "6e294bea-7b40-452e-b272-92c9bea69bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:53.841100 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.841071 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca" pod "6e294bea-7b40-452e-b272-92c9bea69bc4" (UID: "6e294bea-7b40-452e-b272-92c9bea69bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:36:53.841213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.841104 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm" pod "6e294bea-7b40-452e-b272-92c9bea69bc4" (UID: "6e294bea-7b40-452e-b272-92c9bea69bc4"). InnerVolumeSpecName "kube-api-access-996hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:53.940080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.939994 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-996hm\" (UniqueName: \"kubernetes.io/projected/6e294bea-7b40-452e-b272-92c9bea69bc4-kube-api-access-996hm\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:53.940080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.940027 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:53.940080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:53.940038 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e294bea-7b40-452e-b272-92c9bea69bc4-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:54.639186 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:54.639159 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" Oct 30 19:36:54.639359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:54.639201 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5" event={"ID":"6e294bea-7b40-452e-b272-92c9bea69bc4","Type":"ContainerDied","Data":"ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75"} Oct 30 19:36:54.639359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:54.639252 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce847d9c9f496e798157dc09d662df810780f451aad2e63f2dfdd87edfac2e75" Oct 30 19:36:55.952317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:55.952282 6907 scope.go:122] "RemoveContainer" containerID="f260c1dece56535675a49f15d786c9fccbbc2225d867ac5afc55c01d2890dfc7" Oct 30 19:36:55.965090 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:55.965072 6907 scope.go:122] "RemoveContainer" containerID="f88179c9a08e8c8c7b573aee756b2a3eaf04d15d3a2c4574672e0c0ef2198390" Oct 30 19:36:55.974158 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:55.974137 6907 scope.go:122] "RemoveContainer" containerID="68ae856e8792f52ec7afe531abda374b748b809549e86ccf54e03d41cfcae0e0" Oct 30 19:36:55.982864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:55.982837 6907 scope.go:122] "RemoveContainer" containerID="f7904b84061a220a437c850b0021e5fe0cb7bab07341586d6e7a987eed8f5ff9" Oct 30 19:36:55.991300 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:36:55.991282 6907 scope.go:122] "RemoveContainer" containerID="54bd8eba780a1c5d5060b0643278745fe759ae17570b82ce1eeb2f83696f4cfc" Oct 30 19:37:00.025187 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:00.025148 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg"] Oct 30 19:37:00.027662 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:00.027637 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89a9b580-7mmhg"] Oct 30 19:37:00.149843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:00.149809 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f71dcd-6168-411f-8058-b2b0d2d5d774" path="/var/lib/kubelet/pods/61f71dcd-6168-411f-8058-b2b0d2d5d774/volumes" Oct 30 19:37:01.330283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:01.330249 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:06.249648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:06.249609 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:06.250332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:06.250296 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:06.262990 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:06.262941 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:07.735720 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:37:07.735677 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:37:09.028287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.028245 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv"] Oct 30 19:37:09.034203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.034171 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d7ee3c5-495tv"] Oct 30 19:37:09.045818 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.045785 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh"] Oct 30 19:37:09.046318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.046295 6907 cpu_manager.go:414] "containerMap: removing container" podUID="6e294bea-7b40-452e-b272-92c9bea69bc4" containerName="revoke-keys" Oct 30 19:37:09.046318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.046314 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e294bea-7b40-452e-b272-92c9bea69bc4" containerName="revoke-keys" Oct 30 19:37:09.046521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.046407 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e294bea-7b40-452e-b272-92c9bea69bc4" containerName="revoke-keys" Oct 30 19:37:09.051880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.051864 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.056146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.056120 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:09.056479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.056320 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:09.056479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.056375 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:37:09.056479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.056435 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:09.058292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.058268 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh"] Oct 30 19:37:09.142077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.142038 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.142284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.142086 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htr54\" (UniqueName: \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.142284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.142172 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.243474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.243417 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.243682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.243486 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htr54\" (UniqueName: \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.243682 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.243578 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.244265 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.244240 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.245687 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.245666 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.252721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.252699 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htr54\" (UniqueName: \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\") pod \"maas-api-revoke-sub-79805ca9-5fqsh\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.370386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.370294 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:09.655390 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.655362 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh"] Oct 30 19:37:09.657208 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:37:09.657179 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb792789_4752_4353_8c11_180dcecdf3f3.slice/crio-54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac WatchSource:0}: Error finding container 54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac: Status 404 returned error can't find the container with id 54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac Oct 30 19:37:09.780695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:09.780655 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" event={"ID":"cb792789-4752-4353-8c11-180dcecdf3f3","Type":"ContainerStarted","Data":"54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac"} Oct 30 19:37:10.151139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:10.151104 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0addc13-c7eb-40b3-9177-f1cc8b94e0d3" path="/var/lib/kubelet/pods/a0addc13-c7eb-40b3-9177-f1cc8b94e0d3/volumes" Oct 30 19:37:10.788681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:10.788647 6907 generic.go:356] "Generic (PLEG): container finished" podID="cb792789-4752-4353-8c11-180dcecdf3f3" containerID="2fd5928f820be4ad75a324324c4d423385b24bec1a64980521780bb9e1b79de2" exitCode=0 Oct 30 19:37:10.788883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:10.788729 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" event={"ID":"cb792789-4752-4353-8c11-180dcecdf3f3","Type":"ContainerDied","Data":"2fd5928f820be4ad75a324324c4d423385b24bec1a64980521780bb9e1b79de2"} Oct 30 19:37:11.935095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:11.935070 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:12.073061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.072973 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\" (UniqueName: \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\") pod \"cb792789-4752-4353-8c11-180dcecdf3f3\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " Oct 30 19:37:12.073061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.073034 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\") pod \"cb792789-4752-4353-8c11-180dcecdf3f3\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " Oct 30 19:37:12.073266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.073109 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\") pod \"cb792789-4752-4353-8c11-180dcecdf3f3\" (UID: \"cb792789-4752-4353-8c11-180dcecdf3f3\") " Oct 30 19:37:12.073377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.073340 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca" pod "cb792789-4752-4353-8c11-180dcecdf3f3" (UID: "cb792789-4752-4353-8c11-180dcecdf3f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:12.075106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.075086 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca" pod "cb792789-4752-4353-8c11-180dcecdf3f3" (UID: "cb792789-4752-4353-8c11-180dcecdf3f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:12.075219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.075204 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54" pod "cb792789-4752-4353-8c11-180dcecdf3f3" (UID: "cb792789-4752-4353-8c11-180dcecdf3f3"). InnerVolumeSpecName "kube-api-access-htr54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:12.174509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.174469 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:12.174509 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.174504 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htr54\" (UniqueName: \"kubernetes.io/projected/cb792789-4752-4353-8c11-180dcecdf3f3-kube-api-access-htr54\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:12.174722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.174517 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb792789-4752-4353-8c11-180dcecdf3f3-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:12.802801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.802767 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" event={"ID":"cb792789-4752-4353-8c11-180dcecdf3f3","Type":"ContainerDied","Data":"54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac"} Oct 30 19:37:12.802980 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.802806 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh" Oct 30 19:37:12.802980 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:12.802819 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54cbd734aa8b6ed756ec1f699d2004d1d9676a8f66eca09c9f2d85237a222bac" Oct 30 19:37:19.933413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:19.933369 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:21.025856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:21.025816 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66"] Oct 30 19:37:21.030046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:21.030000 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk"] Oct 30 19:37:21.032631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:21.032587 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d921ed-wbx66"] Oct 30 19:37:21.034813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:21.034791 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22b0f0a-mggpk"] Oct 30 19:37:22.154201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:22.154162 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e06ff31-d093-4a83-ac9f-13875a3d6e6c" path="/var/lib/kubelet/pods/7e06ff31-d093-4a83-ac9f-13875a3d6e6c/volumes" Oct 30 19:37:22.155436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:22.155409 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d336c29f-9a58-4372-b615-225c9eb1fb4c" path="/var/lib/kubelet/pods/d336c29f-9a58-4372-b615-225c9eb1fb4c/volumes" Oct 30 19:37:23.759467 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:23.759308 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:26.821256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.821212 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf"] Oct 30 19:37:26.821775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.821709 6907 cpu_manager.go:414] "containerMap: removing container" podUID="cb792789-4752-4353-8c11-180dcecdf3f3" containerName="revoke-keys" Oct 30 19:37:26.821775 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.821722 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb792789-4752-4353-8c11-180dcecdf3f3" containerName="revoke-keys" Oct 30 19:37:26.821852 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.821828 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb792789-4752-4353-8c11-180dcecdf3f3" containerName="revoke-keys" Oct 30 19:37:26.826665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.826642 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.831383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.831352 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:37:26.831582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.831384 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:26.831582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.831402 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:26.831582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.831365 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:26.834251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.834221 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf"] Oct 30 19:37:26.884022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.883986 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.884210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.884046 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.884210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.884141 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvj8g\" (UniqueName: \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.985455 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.985416 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.985664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.985516 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.985664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.985562 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvj8g\" (UniqueName: \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.986232 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.986205 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.987770 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.987740 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:26.994297 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:26.994270 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvj8g\" (UniqueName: \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\") pod \"maas-api-revoke-sub-0b7e8edb-nzwkf\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:27.028892 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.028847 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp"] Oct 30 19:37:27.031765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.031736 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm"] Oct 30 19:37:27.034044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.034025 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e899c5-hnsmp"] Oct 30 19:37:27.036151 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.036132 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53c6ea93-vnlnm"] Oct 30 19:37:27.142524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.142424 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:27.420871 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.420845 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf"] Oct 30 19:37:27.422958 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:37:27.422926 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deb8b3e_16e4_4acb_bf14_5b785f2763ec.slice/crio-0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b WatchSource:0}: Error finding container 0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b: Status 404 returned error can't find the container with id 0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b Oct 30 19:37:27.937255 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.937211 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" event={"ID":"8deb8b3e-16e4-4acb-bf14-5b785f2763ec","Type":"ContainerStarted","Data":"585d58089a33f7c073740281f98bf6fb327c5fca310401da41e496e24b88a3d6"} Oct 30 19:37:27.937772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.937274 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" event={"ID":"8deb8b3e-16e4-4acb-bf14-5b785f2763ec","Type":"ContainerStarted","Data":"0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b"} Oct 30 19:37:27.955514 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:27.955434 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" podStartSLOduration=1.955419389 podStartE2EDuration="1.955419389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:27.953983062 +0000 UTC m=+1053.930209005" watchObservedRunningTime="2026-10-30 19:37:27.955419389 +0000 UTC m=+1053.931645330" Oct 30 19:37:28.149146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:28.149067 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188afca3-d1d3-4e7a-ae42-d12ebc99527d" path="/var/lib/kubelet/pods/188afca3-d1d3-4e7a-ae42-d12ebc99527d/volumes" Oct 30 19:37:28.149529 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:28.149515 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7" path="/var/lib/kubelet/pods/cc34afd9-cdfd-4323-b8b3-a8894cfa9ef7/volumes" Oct 30 19:37:28.952103 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:28.952065 6907 generic.go:356] "Generic (PLEG): container finished" podID="8deb8b3e-16e4-4acb-bf14-5b785f2763ec" containerID="585d58089a33f7c073740281f98bf6fb327c5fca310401da41e496e24b88a3d6" exitCode=0 Oct 30 19:37:28.952543 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:28.952127 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" event={"ID":"8deb8b3e-16e4-4acb-bf14-5b785f2763ec","Type":"ContainerDied","Data":"585d58089a33f7c073740281f98bf6fb327c5fca310401da41e496e24b88a3d6"} Oct 30 19:37:30.098861 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.098836 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:30.220142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.220119 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\" (UniqueName: \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\") pod \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " Oct 30 19:37:30.220301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.220184 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\") pod \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " Oct 30 19:37:30.220301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.220238 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\") pod \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\" (UID: \"8deb8b3e-16e4-4acb-bf14-5b785f2763ec\") " Oct 30 19:37:30.220661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.220624 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca" pod "8deb8b3e-16e4-4acb-bf14-5b785f2763ec" (UID: "8deb8b3e-16e4-4acb-bf14-5b785f2763ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:30.222253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.222227 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g" pod "8deb8b3e-16e4-4acb-bf14-5b785f2763ec" (UID: "8deb8b3e-16e4-4acb-bf14-5b785f2763ec"). InnerVolumeSpecName "kube-api-access-lvj8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:30.222253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.222227 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca" pod "8deb8b3e-16e4-4acb-bf14-5b785f2763ec" (UID: "8deb8b3e-16e4-4acb-bf14-5b785f2763ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:30.321725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.321683 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvj8g\" (UniqueName: \"kubernetes.io/projected/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-kube-api-access-lvj8g\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:30.321725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.321718 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:30.321725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.321728 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8deb8b3e-16e4-4acb-bf14-5b785f2763ec-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:30.965681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.965646 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" Oct 30 19:37:30.965681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.965671 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf" event={"ID":"8deb8b3e-16e4-4acb-bf14-5b785f2763ec","Type":"ContainerDied","Data":"0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b"} Oct 30 19:37:30.965898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:30.965724 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b764460d20b5f77b5e600058982bb8669c85a772c3d467f98edcfac2c7c160b" Oct 30 19:37:37.029971 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.029934 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp"] Oct 30 19:37:37.033453 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.033417 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e302d0d0-jjctp"] Oct 30 19:37:37.470332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.470300 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj"] Oct 30 19:37:37.470796 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.470782 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8deb8b3e-16e4-4acb-bf14-5b785f2763ec" containerName="revoke-keys" Oct 30 19:37:37.470848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.470797 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8deb8b3e-16e4-4acb-bf14-5b785f2763ec" containerName="revoke-keys" Oct 30 19:37:37.470896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.470887 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8deb8b3e-16e4-4acb-bf14-5b785f2763ec" containerName="revoke-keys" Oct 30 19:37:37.474412 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.474389 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.481038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.481015 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:37.481188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.481080 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:37.481188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.481110 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:37:37.481188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.481137 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:37.484938 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.484911 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj"] Oct 30 19:37:37.599696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.599662 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.599696 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.599705 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqpw\" (UniqueName: \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.599958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.599777 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.700436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.700396 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.700646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.700470 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqpw\" (UniqueName: \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.700646 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.700512 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.701155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.701133 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.702962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.702933 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.710119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.710097 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqpw\" (UniqueName: \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\") pod \"maas-api-revoke-sub-3d4156f1-ltvqj\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:37.735638 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.735559 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:37.737001 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:37:37.736975 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:37:37.790723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:37.790688 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:38.070297 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:38.070268 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj"] Oct 30 19:37:38.072387 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:37:38.072357 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73c40c6_78b3_4fe3_a362_83d364f0c5a0.slice/crio-d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718 WatchSource:0}: Error finding container d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718: Status 404 returned error can't find the container with id d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718 Oct 30 19:37:38.151252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:38.151215 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5475738-1b37-4a2b-9f3a-0ee4f2eae500" path="/var/lib/kubelet/pods/d5475738-1b37-4a2b-9f3a-0ee4f2eae500/volumes" Oct 30 19:37:39.020277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:39.020239 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" event={"ID":"a73c40c6-78b3-4fe3-a362-83d364f0c5a0","Type":"ContainerStarted","Data":"a81e046e9f66499ffa465e83a51b417dc7e033209a979bed73f64f45b6ab0ec7"} Oct 30 19:37:39.020493 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:39.020303 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" event={"ID":"a73c40c6-78b3-4fe3-a362-83d364f0c5a0","Type":"ContainerStarted","Data":"d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718"} Oct 30 19:37:39.037217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:39.037169 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" podStartSLOduration=2.037155047 podStartE2EDuration="2.037155047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:39.036225199 +0000 UTC m=+1065.012451144" watchObservedRunningTime="2026-10-30 19:37:39.037155047 +0000 UTC m=+1065.013380991" Oct 30 19:37:40.028968 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:40.028932 6907 generic.go:356] "Generic (PLEG): container finished" podID="a73c40c6-78b3-4fe3-a362-83d364f0c5a0" containerID="a81e046e9f66499ffa465e83a51b417dc7e033209a979bed73f64f45b6ab0ec7" exitCode=0 Oct 30 19:37:40.029392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:40.029023 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" event={"ID":"a73c40c6-78b3-4fe3-a362-83d364f0c5a0","Type":"ContainerDied","Data":"a81e046e9f66499ffa465e83a51b417dc7e033209a979bed73f64f45b6ab0ec7"} Oct 30 19:37:41.177756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.177729 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:41.336426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.336323 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\" (UniqueName: \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\") pod \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " Oct 30 19:37:41.336426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.336385 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\") pod \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " Oct 30 19:37:41.336712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.336540 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\") pod \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\" (UID: \"a73c40c6-78b3-4fe3-a362-83d364f0c5a0\") " Oct 30 19:37:41.336815 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.336792 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca" pod "a73c40c6-78b3-4fe3-a362-83d364f0c5a0" (UID: "a73c40c6-78b3-4fe3-a362-83d364f0c5a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:41.338595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.338564 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca" pod "a73c40c6-78b3-4fe3-a362-83d364f0c5a0" (UID: "a73c40c6-78b3-4fe3-a362-83d364f0c5a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:41.338683 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.338655 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw" pod "a73c40c6-78b3-4fe3-a362-83d364f0c5a0" (UID: "a73c40c6-78b3-4fe3-a362-83d364f0c5a0"). InnerVolumeSpecName "kube-api-access-vvqpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:41.438277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.438236 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:41.438277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.438273 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:41.438277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:41.438284 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvqpw\" (UniqueName: \"kubernetes.io/projected/a73c40c6-78b3-4fe3-a362-83d364f0c5a0-kube-api-access-vvqpw\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:42.044874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:42.044834 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" Oct 30 19:37:42.044874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:42.044859 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj" event={"ID":"a73c40c6-78b3-4fe3-a362-83d364f0c5a0","Type":"ContainerDied","Data":"d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718"} Oct 30 19:37:42.045086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:42.044910 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d653f255aaa3af9f07871c5c6b329f6730f746b4e1a8b63ecc8ef3481249c718" Oct 30 19:37:48.445150 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:48.445114 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:51.239469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:51.238570 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-bcxng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:37:51.239469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:51.238659 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-bcxng" podUID="d29f3ff5-991d-4ca7-af41-a592d6e767da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:37:52.772465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:52.769664 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:37:55.029745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.029707 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p"] Oct 30 19:37:55.035201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.035173 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03ff582-l9c9p"] Oct 30 19:37:55.500264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.500227 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj"] Oct 30 19:37:55.506593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.506563 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ltvqj"] Oct 30 19:37:55.539531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.539494 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq"] Oct 30 19:37:55.539999 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.539984 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a73c40c6-78b3-4fe3-a362-83d364f0c5a0" containerName="revoke-keys" Oct 30 19:37:55.540059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.540000 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a73c40c6-78b3-4fe3-a362-83d364f0c5a0" containerName="revoke-keys" Oct 30 19:37:55.540110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.540100 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a73c40c6-78b3-4fe3-a362-83d364f0c5a0" containerName="revoke-keys" Oct 30 19:37:55.543511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.543493 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.556298 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.556278 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:37:55.556538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.556523 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:37:55.560407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.560384 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq"] Oct 30 19:37:55.563163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.563140 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:37:55.563268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.563237 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:37:55.656593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.656559 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.656593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.656596 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.656841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.656740 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj49\" (UniqueName: \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.758183 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.758089 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj49\" (UniqueName: \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.758375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.758199 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.758375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.758230 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.758877 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.758853 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.760711 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.760686 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.770640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.770615 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj49\" (UniqueName: \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\") pod \"maas-api-revoke-sub-3d4156f1-ntnrq\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:55.856487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:55.856424 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:56.034716 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.034635 6907 scope.go:122] "RemoveContainer" containerID="8c3e2d1e4e94796e4284c32817d56d90e99771034b8fd420a26b6e90acf7c02b" Oct 30 19:37:56.044530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.044504 6907 scope.go:122] "RemoveContainer" containerID="daabf3c0bdde098dc6c5f19845caddaeb62bdac4e71f60c0aec52b245ef0c0cf" Oct 30 19:37:56.054156 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.054130 6907 scope.go:122] "RemoveContainer" containerID="2ac5e50624446c891defc94864beb99537c332334bd2baed41f1919c18c36d8b" Oct 30 19:37:56.063044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.063024 6907 scope.go:122] "RemoveContainer" containerID="4871f46ebbbdeb77fd607641fcdfabe486de7a78cc7f8909a08e35985db5c7fc" Oct 30 19:37:56.072020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.072002 6907 scope.go:122] "RemoveContainer" containerID="bb94f1b2f0393dff234ea2440c25951fe8efc26d77130fa608c109dba2548384" Oct 30 19:37:56.147584 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:37:56.147549 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15cf90e_f7f6_49ea_b446_b9b8566e38bc.slice/crio-08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0 WatchSource:0}: Error finding container 08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0: Status 404 returned error can't find the container with id 08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0 Oct 30 19:37:56.151457 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.151420 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a73c40c6-78b3-4fe3-a362-83d364f0c5a0" path="/var/lib/kubelet/pods/a73c40c6-78b3-4fe3-a362-83d364f0c5a0/volumes" Oct 30 19:37:56.151840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.151825 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f77dd963-bc60-424e-85fb-a2035f2dcad0" path="/var/lib/kubelet/pods/f77dd963-bc60-424e-85fb-a2035f2dcad0/volumes" Oct 30 19:37:56.152118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.152105 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq"] Oct 30 19:37:56.195168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:56.195139 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" event={"ID":"b15cf90e-f7f6-49ea-b446-b9b8566e38bc","Type":"ContainerStarted","Data":"08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0"} Oct 30 19:37:57.202845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:57.202750 6907 generic.go:356] "Generic (PLEG): container finished" podID="b15cf90e-f7f6-49ea-b446-b9b8566e38bc" containerID="cece2f6aa819b1772b08b89cedd264487ce497d8c32285cd4ea5332eb777ed05" exitCode=0 Oct 30 19:37:57.203225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:57.202834 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" event={"ID":"b15cf90e-f7f6-49ea-b446-b9b8566e38bc","Type":"ContainerDied","Data":"cece2f6aa819b1772b08b89cedd264487ce497d8c32285cd4ea5332eb777ed05"} Oct 30 19:37:58.348569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.348542 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:58.384138 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.384110 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\") pod \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " Oct 30 19:37:58.384335 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.384169 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\") pod \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " Oct 30 19:37:58.384335 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.384202 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\" (UniqueName: \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\") pod \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\" (UID: \"b15cf90e-f7f6-49ea-b446-b9b8566e38bc\") " Oct 30 19:37:58.384572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.384546 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca" pod "b15cf90e-f7f6-49ea-b446-b9b8566e38bc" (UID: "b15cf90e-f7f6-49ea-b446-b9b8566e38bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:37:58.386285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.386259 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca" pod "b15cf90e-f7f6-49ea-b446-b9b8566e38bc" (UID: "b15cf90e-f7f6-49ea-b446-b9b8566e38bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:58.386437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.386354 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49" pod "b15cf90e-f7f6-49ea-b446-b9b8566e38bc" (UID: "b15cf90e-f7f6-49ea-b446-b9b8566e38bc"). InnerVolumeSpecName "kube-api-access-drj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:58.486048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.486017 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.486048 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.486047 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:58.486246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:58.486058 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj49\" (UniqueName: \"kubernetes.io/projected/b15cf90e-f7f6-49ea-b446-b9b8566e38bc-kube-api-access-drj49\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:59.217438 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:59.217406 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" Oct 30 19:37:59.217633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:59.217433 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq" event={"ID":"b15cf90e-f7f6-49ea-b446-b9b8566e38bc","Type":"ContainerDied","Data":"08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0"} Oct 30 19:37:59.217633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:37:59.217507 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08960c2de5608d04d20006cf4f28cacf34cb9282ab98db2ce7bdc39fcd2141d0" Oct 30 19:38:03.027008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:03.026970 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm"] Oct 30 19:38:03.031778 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:03.031743 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be0964eb-gpdkm"] Oct 30 19:38:04.029357 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:04.029324 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl"] Oct 30 19:38:04.032244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:04.032215 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84159f60-8djkl"] Oct 30 19:38:04.150506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:04.150474 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd" path="/var/lib/kubelet/pods/1b09d63d-85e2-4315-8c6c-4e7a59bfe4fd/volumes" Oct 30 19:38:04.150920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:04.150906 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3980a1f9-5137-4289-b097-2ec6f1023536" path="/var/lib/kubelet/pods/3980a1f9-5137-4289-b097-2ec6f1023536/volumes" Oct 30 19:38:07.737911 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:38:07.737875 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:38:13.456767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:13.456726 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:38:16.181469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.180685 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:16.181469 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.180771 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:16.405468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.404919 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:16.405468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.404994 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:16.642592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.642550 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:16.642912 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:16.642879 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:17.687549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.687512 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v"] Oct 30 19:38:17.687986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.687970 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b15cf90e-f7f6-49ea-b446-b9b8566e38bc" containerName="revoke-keys" Oct 30 19:38:17.688037 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.687988 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15cf90e-f7f6-49ea-b446-b9b8566e38bc" containerName="revoke-keys" Oct 30 19:38:17.688093 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.688084 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15cf90e-f7f6-49ea-b446-b9b8566e38bc" containerName="revoke-keys" Oct 30 19:38:17.691435 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.691409 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.698792 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.698756 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:38:17.698939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.698884 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:38:17.702312 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.702287 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v"] Oct 30 19:38:17.780137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.780097 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:17.780137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.780131 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:17.826015 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.825976 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.826201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.826099 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s252v\" (UniqueName: \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.826201 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.826143 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.926945 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.926905 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.927157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.926954 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s252v\" (UniqueName: \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.927157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.926977 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.927745 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.927719 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.929286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.929265 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:17.936353 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:17.936327 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s252v\" (UniqueName: \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\") pod \"maas-api-revoke-sub-e2fc9c5b-7pf5v\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:18.007197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:18.007167 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:18.288678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:18.288648 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v"] Oct 30 19:38:18.290399 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:38:18.290365 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b24381_6516_49a4_aa32_835be3d90b13.slice/crio-b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a WatchSource:0}: Error finding container b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a: Status 404 returned error can't find the container with id b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a Oct 30 19:38:18.375913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:18.375881 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" event={"ID":"17b24381-6516-49a4-aa32-835be3d90b13","Type":"ContainerStarted","Data":"b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a"} Oct 30 19:38:19.383974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:19.383888 6907 generic.go:356] "Generic (PLEG): container finished" podID="17b24381-6516-49a4-aa32-835be3d90b13" containerID="61dad6a82398ebb1fbef7d42691b6516acd7302ec5dbaf142bf6f508c1c1e4ba" exitCode=0 Oct 30 19:38:19.384359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:19.383981 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" event={"ID":"17b24381-6516-49a4-aa32-835be3d90b13","Type":"ContainerDied","Data":"61dad6a82398ebb1fbef7d42691b6516acd7302ec5dbaf142bf6f508c1c1e4ba"} Oct 30 19:38:20.533107 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.533081 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:20.654698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.654613 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\" (UniqueName: \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\") pod \"17b24381-6516-49a4-aa32-835be3d90b13\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " Oct 30 19:38:20.654853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.654717 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\") pod \"17b24381-6516-49a4-aa32-835be3d90b13\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " Oct 30 19:38:20.654853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.654805 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\") pod \"17b24381-6516-49a4-aa32-835be3d90b13\" (UID: \"17b24381-6516-49a4-aa32-835be3d90b13\") " Oct 30 19:38:20.655148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.655115 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca" pod "17b24381-6516-49a4-aa32-835be3d90b13" (UID: "17b24381-6516-49a4-aa32-835be3d90b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:20.656763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.656739 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v" pod "17b24381-6516-49a4-aa32-835be3d90b13" (UID: "17b24381-6516-49a4-aa32-835be3d90b13"). InnerVolumeSpecName "kube-api-access-s252v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:20.656868 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.656779 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca" pod "17b24381-6516-49a4-aa32-835be3d90b13" (UID: "17b24381-6516-49a4-aa32-835be3d90b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:20.756325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.756283 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:20.756325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.756317 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17b24381-6516-49a4-aa32-835be3d90b13-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:20.756325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:20.756327 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s252v\" (UniqueName: \"kubernetes.io/projected/17b24381-6516-49a4-aa32-835be3d90b13-kube-api-access-s252v\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:21.399316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:21.399287 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" Oct 30 19:38:21.399559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:21.399281 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v" event={"ID":"17b24381-6516-49a4-aa32-835be3d90b13","Type":"ContainerDied","Data":"b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a"} Oct 30 19:38:21.399559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:21.399424 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b55dfe61c13a129432e646e3684528b3bedb40e73426564cf2010de6b560667a" Oct 30 19:38:27.909164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.909122 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv"] Oct 30 19:38:27.909753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.909736 6907 cpu_manager.go:414] "containerMap: removing container" podUID="17b24381-6516-49a4-aa32-835be3d90b13" containerName="revoke-keys" Oct 30 19:38:27.909805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.909755 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17b24381-6516-49a4-aa32-835be3d90b13" containerName="revoke-keys" Oct 30 19:38:27.909872 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.909864 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="17b24381-6516-49a4-aa32-835be3d90b13" containerName="revoke-keys" Oct 30 19:38:27.913267 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.913249 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:27.919301 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.919270 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:38:27.919511 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.919397 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:38:27.919650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.919632 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:38:27.919726 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.919687 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:38:27.922207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:27.921959 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv"] Oct 30 19:38:28.027739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.027700 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8dn\" (UniqueName: \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.027967 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.027755 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.028028 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.027957 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.129032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.128997 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8dn\" (UniqueName: \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.129247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.129056 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.129247 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.129225 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.129874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.129848 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.131537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.131508 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.138940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.138918 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8dn\" (UniqueName: \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\") pod \"maas-api-revoke-sub-5a29efb3-48tqv\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.230272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.230242 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:28.511806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.511780 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv"] Oct 30 19:38:28.514438 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:38:28.514408 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8660d443_c074_4d6b_977e_e4a6fbeac905.slice/crio-7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072 WatchSource:0}: Error finding container 7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072: Status 404 returned error can't find the container with id 7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072 Oct 30 19:38:28.533983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:28.533956 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:38:29.028088 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:29.028041 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn"] Oct 30 19:38:29.034759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:29.034718 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578da981-58zdn"] Oct 30 19:38:29.459069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:29.459022 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" event={"ID":"8660d443-c074-4d6b-977e-e4a6fbeac905","Type":"ContainerStarted","Data":"c5c897e38227dc0db5feceeab150bd01d2c31c518fec8b6bbc2a70046ed0d628"} Oct 30 19:38:29.459414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:29.459097 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" event={"ID":"8660d443-c074-4d6b-977e-e4a6fbeac905","Type":"ContainerStarted","Data":"7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072"} Oct 30 19:38:29.491596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:29.491519 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" podStartSLOduration=2.491496845 podStartE2EDuration="2.491496845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:29.488467441 +0000 UTC m=+1115.464693379" watchObservedRunningTime="2026-10-30 19:38:29.491496845 +0000 UTC m=+1115.467722789" Oct 30 19:38:30.187277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:30.187233 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6701363-dcd9-484f-9f6e-aa7d40db8774" path="/var/lib/kubelet/pods/d6701363-dcd9-484f-9f6e-aa7d40db8774/volumes" Oct 30 19:38:30.493437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:30.493394 6907 generic.go:356] "Generic (PLEG): container finished" podID="8660d443-c074-4d6b-977e-e4a6fbeac905" containerID="c5c897e38227dc0db5feceeab150bd01d2c31c518fec8b6bbc2a70046ed0d628" exitCode=0 Oct 30 19:38:30.493857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:30.493832 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" event={"ID":"8660d443-c074-4d6b-977e-e4a6fbeac905","Type":"ContainerDied","Data":"c5c897e38227dc0db5feceeab150bd01d2c31c518fec8b6bbc2a70046ed0d628"} Oct 30 19:38:31.173481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.172568 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:31.173481 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.172646 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:31.386593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.386552 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:31.387213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.387180 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:31.650075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.649568 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:38:31.650075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.649639 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:38:31.778340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.778274 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:31.825256 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.825218 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\") pod \"8660d443-c074-4d6b-977e-e4a6fbeac905\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " Oct 30 19:38:31.825430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.825385 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\" (UniqueName: \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\") pod \"8660d443-c074-4d6b-977e-e4a6fbeac905\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " Oct 30 19:38:31.825512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.825493 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\") pod \"8660d443-c074-4d6b-977e-e4a6fbeac905\" (UID: \"8660d443-c074-4d6b-977e-e4a6fbeac905\") " Oct 30 19:38:31.825839 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.825812 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca" pod "8660d443-c074-4d6b-977e-e4a6fbeac905" (UID: "8660d443-c074-4d6b-977e-e4a6fbeac905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:38:31.827906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.827870 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca" pod "8660d443-c074-4d6b-977e-e4a6fbeac905" (UID: "8660d443-c074-4d6b-977e-e4a6fbeac905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:31.827906 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.827899 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn" pod "8660d443-c074-4d6b-977e-e4a6fbeac905" (UID: "8660d443-c074-4d6b-977e-e4a6fbeac905"). InnerVolumeSpecName "kube-api-access-7s8dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:31.926937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.926905 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8dn\" (UniqueName: \"kubernetes.io/projected/8660d443-c074-4d6b-977e-e4a6fbeac905-kube-api-access-7s8dn\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:31.926937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.926934 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:31.926937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:31.926946 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8660d443-c074-4d6b-977e-e4a6fbeac905-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:32.527356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:32.527319 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" event={"ID":"8660d443-c074-4d6b-977e-e4a6fbeac905","Type":"ContainerDied","Data":"7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072"} Oct 30 19:38:32.527898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:32.527359 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv" Oct 30 19:38:32.527898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:32.527374 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aebfcea54ee28e9d9f5c249ac3640db6849b6db62515d0a691df04301631072" Oct 30 19:38:37.736637 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:38:37.736590 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:38:38.737021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:38.736984 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:38:43.635162 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:43.635124 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq"] Oct 30 19:38:43.642431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:43.642395 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ntnrq"] Oct 30 19:38:44.149928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:44.149896 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15cf90e-f7f6-49ea-b446-b9b8566e38bc" path="/var/lib/kubelet/pods/b15cf90e-f7f6-49ea-b446-b9b8566e38bc/volumes" Oct 30 19:38:56.117047 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.117011 6907 scope.go:122] "RemoveContainer" containerID="a4a63b5ef0dd7617b0adc91a311d178ca436eb4425d4111b710c9552d4702824" Oct 30 19:38:56.126957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.126923 6907 scope.go:122] "RemoveContainer" containerID="b356a955fff2015edb7b341ac1770cb69c3d35693498c59b9042f0e45ccaef94" Oct 30 19:38:56.136525 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.136505 6907 scope.go:122] "RemoveContainer" containerID="04144c21243c677e397adf2f901f40a9e8d458311a5bd04158c3d2fba2b2874c" Oct 30 19:38:56.149282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.149258 6907 scope.go:122] "RemoveContainer" containerID="23f089d4a49145a2d2e655e3d090e0ade8097058fc1218cbfb7c28e007b39cd6" Oct 30 19:38:56.159176 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.159151 6907 scope.go:122] "RemoveContainer" containerID="0d234574341e68b06cb3faae49dbeba21f519b29f38587134d92d8c58457149f" Oct 30 19:38:56.167832 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.167813 6907 scope.go:122] "RemoveContainer" containerID="0d2759ffbc2bd4028f470fa8aa739c948a2278f9886d14c2067543d120f1bea0" Oct 30 19:38:56.176518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.176491 6907 scope.go:122] "RemoveContainer" containerID="b5ba594d360b56cbbc769b4179444447d5bd94e16e3f675d83c84d1eafae970b" Oct 30 19:38:56.185069 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.185047 6907 scope.go:122] "RemoveContainer" containerID="df411df8404e91172ef297afb9eeca3057f9947c75aefb2d6f948c2acba8b1b0" Oct 30 19:38:56.193843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.193814 6907 scope.go:122] "RemoveContainer" containerID="48e8356e198c8d90a02b730c18ff5f346c602887cea63d67ec0d9f2deffbd8be" Oct 30 19:38:56.202012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.201992 6907 scope.go:122] "RemoveContainer" containerID="9eb77fc2c55cb0a2999ba6d10ec2b5a8123a19e2f5627d7a84f94b367ac75616" Oct 30 19:38:56.210032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.210015 6907 scope.go:122] "RemoveContainer" containerID="bd78bfa57a14f8c3aefc26517cb45d529c70478da6f268222708927ea1857547" Oct 30 19:38:56.218620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:56.218603 6907 scope.go:122] "RemoveContainer" containerID="7fa92c72597ab05ebc76509e752b46a2857cec7c4a685857f234e025ed90b00e" Oct 30 19:38:59.021612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.021576 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:38:59.022089 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.022074 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8660d443-c074-4d6b-977e-e4a6fbeac905" containerName="revoke-keys" Oct 30 19:38:59.022132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.022091 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8660d443-c074-4d6b-977e-e4a6fbeac905" containerName="revoke-keys" Oct 30 19:38:59.022794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.022745 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8660d443-c074-4d6b-977e-e4a6fbeac905" containerName="revoke-keys" Oct 30 19:38:59.039023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.038990 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.039908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.039878 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:38:59.050392 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.050360 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-88ec2c-openshift-default-dockercfg-9d2jc\"" Oct 30 19:38:59.196549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196505 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196549 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196554 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196573 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196690 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwgj\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196757 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196780 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196916 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196822 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196916 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196880 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.196916 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.196899 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298181 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298225 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298246 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298399 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwgj\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298524 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298577 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298645 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298682 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298733 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298763 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.298809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.298764 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.299084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.299059 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.299150 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.299062 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.299471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.299425 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.301100 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.301079 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.301221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.301204 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.310814 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.310789 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.310957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.310835 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwgj\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\") pod \"e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.354507 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.354472 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:38:59.655833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.655805 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:38:59.657951 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:38:59.657918 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceac5dfc_dc74_4b6e_aa81_e025867e4920.slice/crio-951e956f90afbd817da25f51900de43072d4bd5fcd62cb943e28beff6009403e WatchSource:0}: Error finding container 951e956f90afbd817da25f51900de43072d4bd5fcd62cb943e28beff6009403e: Status 404 returned error can't find the container with id 951e956f90afbd817da25f51900de43072d4bd5fcd62cb943e28beff6009403e Oct 30 19:38:59.660157 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.660126 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:59.660225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.660196 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:59.660263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.660225 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:38:59.716793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:38:59.716761 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerStarted","Data":"951e956f90afbd817da25f51900de43072d4bd5fcd62cb943e28beff6009403e"} Oct 30 19:39:00.724651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:00.724614 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerStarted","Data":"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e"} Oct 30 19:39:00.745472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:00.745407 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podStartSLOduration=2.745392103 podStartE2EDuration="2.745392103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:00.745360735 +0000 UTC m=+1146.721586691" watchObservedRunningTime="2026-10-30 19:39:00.745392103 +0000 UTC m=+1146.721618047" Oct 30 19:39:01.355546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:01.355505 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:01.360779 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:01.360752 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:01.730559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:01.730525 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:01.731734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:01.731712 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:07.035253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:07.035218 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:39:07.736345 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:39:07.736305 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:39:16.953223 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:16.953171 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:36338->10.134.0.138:15021: read: connection reset by peer" start-of-body= Oct 30 19:39:16.953649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:16.953274 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": read tcp 10.134.0.2:36338->10.134.0.138:15021: read: connection reset by peer" Oct 30 19:39:17.524965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.524926 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:39:17.866570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.866484 6907 generic.go:356] "Generic (PLEG): container finished" podID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerID="4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e" exitCode=0 Oct 30 19:39:17.866718 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.866562 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerDied","Data":"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e"} Oct 30 19:39:17.866984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.866970 6907 scope.go:122] "RemoveContainer" containerID="4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e" Oct 30 19:39:17.868350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.868322 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:17.868437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.868397 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:17.868437 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:17.868430 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:39:18.033235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:18.033189 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n"] Oct 30 19:39:18.037842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:18.037804 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4z25n"] Oct 30 19:39:18.152117 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:18.152017 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc27de4a-58b6-4771-a4e9-d119fd177944" path="/var/lib/kubelet/pods/fc27de4a-58b6-4771-a4e9-d119fd177944/volumes" Oct 30 19:39:18.874513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:18.874474 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerStarted","Data":"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0"} Oct 30 19:39:19.355388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:19.355348 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:19.355865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:19.355655 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:39:19.355865 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:19.355720 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:39:20.357876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:20.357832 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:39:20.358535 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:20.358501 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:39:21.356045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:21.356008 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:39:21.356350 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:21.356324 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:39:22.355601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:22.355553 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:39:22.356045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:22.355633 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:39:23.355867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:23.355811 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 19:39:23.356293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:23.355899 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 19:39:24.360911 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:24.360877 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:24.361658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:24.361629 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:24.363003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:24.362977 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:39:32.312512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:32.312473 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:39:37.737117 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:39:37.737024 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:39:44.670806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.670769 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5"] Oct 30 19:39:44.674636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.674615 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.680228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.680200 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:39:44.680346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.680247 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:39:44.681072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.681053 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:39:44.681072 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.681061 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:39:44.687312 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.687286 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5"] Oct 30 19:39:44.765612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.765572 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.765803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.765724 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.765856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.765806 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbgz\" (UniqueName: \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.866793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.866759 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbgz\" (UniqueName: \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.867004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.866825 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.867004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.866898 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.867542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.867519 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.869235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.869207 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.875586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.875557 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbgz\" (UniqueName: \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\") pod \"maas-api-revoke-sub-cd80ce20-tscj5\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:44.990599 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:44.990563 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:45.276430 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:45.276392 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5"] Oct 30 19:39:45.278110 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:39:45.278078 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a98dd22_4dd2_4b8b_8728_b88323a5a1f4.slice/crio-3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5 WatchSource:0}: Error finding container 3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5: Status 404 returned error can't find the container with id 3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5 Oct 30 19:39:46.049864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:46.049831 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" event={"ID":"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4","Type":"ContainerStarted","Data":"d6fe72851be0b0f3cd88ba38b5dc624812e9c5830c51f03afdccba47c4745b53"} Oct 30 19:39:46.050211 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:46.049885 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" event={"ID":"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4","Type":"ContainerStarted","Data":"3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5"} Oct 30 19:39:46.073080 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:46.073022 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" podStartSLOduration=2.073008995 podStartE2EDuration="2.073008995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:46.071211228 +0000 UTC m=+1192.047437173" watchObservedRunningTime="2026-10-30 19:39:46.073008995 +0000 UTC m=+1192.049234939" Oct 30 19:39:47.057010 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:47.056976 6907 generic.go:356] "Generic (PLEG): container finished" podID="0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" containerID="d6fe72851be0b0f3cd88ba38b5dc624812e9c5830c51f03afdccba47c4745b53" exitCode=0 Oct 30 19:39:47.057433 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:47.057065 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" event={"ID":"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4","Type":"ContainerDied","Data":"d6fe72851be0b0f3cd88ba38b5dc624812e9c5830c51f03afdccba47c4745b53"} Oct 30 19:39:48.207661 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.207635 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:48.303194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.303159 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\") pod \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " Oct 30 19:39:48.303377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.303223 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\") pod \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " Oct 30 19:39:48.303377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.303334 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\" (UniqueName: \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\") pod \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\" (UID: \"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4\") " Oct 30 19:39:48.303616 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.303592 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca" pod "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" (UID: "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:48.303779 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.303762 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:48.305394 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.305365 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca" pod "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" (UID: "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:48.305475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.305420 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz" pod "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" (UID: "0a98dd22-4dd2-4b8b-8728-b88323a5a1f4"). InnerVolumeSpecName "kube-api-access-bxbgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:48.404972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.404872 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:48.404972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:48.404917 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbgz\" (UniqueName: \"kubernetes.io/projected/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4-kube-api-access-bxbgz\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:49.073539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:49.073512 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" Oct 30 19:39:49.073539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:49.073516 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5" event={"ID":"0a98dd22-4dd2-4b8b-8728-b88323a5a1f4","Type":"ContainerDied","Data":"3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5"} Oct 30 19:39:49.073763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:49.073572 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5393d5e81f04fd3dbfcdacc860fd521719740d96a040f46158ffee16162eb5" Oct 30 19:39:54.336415 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:54.336378 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:39:54.343554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:54.343525 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:39:54.357465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:54.357419 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:39:54.364540 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:54.364521 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:39:55.718228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.718192 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q"] Oct 30 19:39:55.718699 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.718685 6907 cpu_manager.go:414] "containerMap: removing container" podUID="0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" containerName="revoke-keys" Oct 30 19:39:55.718750 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.718700 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" containerName="revoke-keys" Oct 30 19:39:55.718802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.718790 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" containerName="revoke-keys" Oct 30 19:39:55.722307 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.722286 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.729110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.729073 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:39:55.729243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.729186 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:39:55.729398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.729378 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:39:55.729545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.729480 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:39:55.732524 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.732502 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q"] Oct 30 19:39:55.784766 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.784728 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.784974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.784938 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.785054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.785011 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2drb\" (UniqueName: \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.887981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.887940 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.888218 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.888197 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.888284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.888251 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2drb\" (UniqueName: \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.890550 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.890514 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.890997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.890975 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:55.902188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:55.902160 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2drb\" (UniqueName: \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:56.038421 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.038387 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:56.238812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.238772 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:39:56.292596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.292518 6907 scope.go:122] "RemoveContainer" containerID="9a0d6d3ca9e7d1336c63db06be60a648d27a813d636f02b2a211047953472cfa" Oct 30 19:39:56.303966 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.303943 6907 scope.go:122] "RemoveContainer" containerID="7f36faf402cecb428803a2a5f2315bb91253b8630ea16d03f8356657cadf667d" Oct 30 19:39:56.315272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.315221 6907 scope.go:122] "RemoveContainer" containerID="8d3b193f4af4643729871dd224fa700fc1c8c6e698e3a8c820e5a9516d9986bf" Oct 30 19:39:56.316208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:56.316137 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q"] Oct 30 19:39:56.320825 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:39:56.320797 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8311892a_bbe7_4e4e_a78c_f1666135c919.slice/crio-6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768 WatchSource:0}: Error finding container 6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768: Status 404 returned error can't find the container with id 6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768 Oct 30 19:39:57.125391 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:57.125353 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" event={"ID":"8311892a-bbe7-4e4e-a78c-f1666135c919","Type":"ContainerStarted","Data":"e3ef25b2e693d75209639f7177650cf6d72822ba0c859062717a5e78ea1dc122"} Oct 30 19:39:57.125801 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:57.125432 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" event={"ID":"8311892a-bbe7-4e4e-a78c-f1666135c919","Type":"ContainerStarted","Data":"6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768"} Oct 30 19:39:57.144506 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:57.144427 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" podStartSLOduration=2.144413998 podStartE2EDuration="2.144413998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:57.141472168 +0000 UTC m=+1203.117698106" watchObservedRunningTime="2026-10-30 19:39:57.144413998 +0000 UTC m=+1203.120639936" Oct 30 19:39:58.133936 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:58.133901 6907 generic.go:356] "Generic (PLEG): container finished" podID="8311892a-bbe7-4e4e-a78c-f1666135c919" containerID="e3ef25b2e693d75209639f7177650cf6d72822ba0c859062717a5e78ea1dc122" exitCode=0 Oct 30 19:39:58.134343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:58.133945 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" event={"ID":"8311892a-bbe7-4e4e-a78c-f1666135c919","Type":"ContainerDied","Data":"e3ef25b2e693d75209639f7177650cf6d72822ba0c859062717a5e78ea1dc122"} Oct 30 19:39:59.291409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.291383 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:39:59.434038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.433962 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\" (UniqueName: \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\") pod \"8311892a-bbe7-4e4e-a78c-f1666135c919\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " Oct 30 19:39:59.434038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.434028 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\") pod \"8311892a-bbe7-4e4e-a78c-f1666135c919\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " Oct 30 19:39:59.434289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.434061 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\") pod \"8311892a-bbe7-4e4e-a78c-f1666135c919\" (UID: \"8311892a-bbe7-4e4e-a78c-f1666135c919\") " Oct 30 19:39:59.434477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.434438 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca" pod "8311892a-bbe7-4e4e-a78c-f1666135c919" (UID: "8311892a-bbe7-4e4e-a78c-f1666135c919"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:39:59.436139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.436120 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb" pod "8311892a-bbe7-4e4e-a78c-f1666135c919" (UID: "8311892a-bbe7-4e4e-a78c-f1666135c919"). InnerVolumeSpecName "kube-api-access-w2drb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:39:59.436195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.436118 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca" pod "8311892a-bbe7-4e4e-a78c-f1666135c919" (UID: "8311892a-bbe7-4e4e-a78c-f1666135c919"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:39:59.534879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.534844 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2drb\" (UniqueName: \"kubernetes.io/projected/8311892a-bbe7-4e4e-a78c-f1666135c919-kube-api-access-w2drb\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:59.534879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.534874 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:39:59.534879 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:39:59.534884 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8311892a-bbe7-4e4e-a78c-f1666135c919-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:00.149754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:00.149721 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" Oct 30 19:40:00.150153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:00.150127 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q" event={"ID":"8311892a-bbe7-4e4e-a78c-f1666135c919","Type":"ContainerDied","Data":"6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768"} Oct 30 19:40:00.150266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:00.150177 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ec724189137654f4e7a62148123e78291b1bb1a225a01790c998296aae04768" Oct 30 19:40:06.318007 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:06.317965 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q"] Oct 30 19:40:06.322931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:06.322904 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-88ec2c-f8q4q"] Oct 30 19:40:07.642536 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:07.642496 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:40:07.766279 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:40:07.766239 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:40:08.150164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:08.150129 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8311892a-bbe7-4e4e-a78c-f1666135c919" path="/var/lib/kubelet/pods/8311892a-bbe7-4e4e-a78c-f1666135c919/volumes" Oct 30 19:40:11.049116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:11.049076 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr"] Oct 30 19:40:11.054690 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:11.054656 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8knxr"] Oct 30 19:40:12.150413 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:12.150379 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f19bfa-c8fb-44f3-865c-7b303bde8739" path="/var/lib/kubelet/pods/49f19bfa-c8fb-44f3-865c-7b303bde8739/volumes" Oct 30 19:40:13.515253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:13.515205 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:40:13.515686 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:13.515505 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" containerID="cri-o://65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0" gracePeriod=30 Oct 30 19:40:14.884693 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.884656 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd"] Oct 30 19:40:14.885299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.885279 6907 cpu_manager.go:414] "containerMap: removing container" podUID="8311892a-bbe7-4e4e-a78c-f1666135c919" containerName="revoke-keys" Oct 30 19:40:14.885352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.885303 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8311892a-bbe7-4e4e-a78c-f1666135c919" containerName="revoke-keys" Oct 30 19:40:14.885434 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.885423 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="8311892a-bbe7-4e4e-a78c-f1666135c919" containerName="revoke-keys" Oct 30 19:40:14.888528 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.888511 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:14.892955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.892934 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:40:14.892955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.892934 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:14.893108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.892977 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:40:14.893108 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.893074 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:40:14.911606 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:14.911576 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd"] Oct 30 19:40:15.001063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.001026 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.001238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.001096 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkvf\" (UniqueName: \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.001279 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.001241 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.102809 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.102770 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkvf\" (UniqueName: \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.103009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.102891 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.103009 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.102965 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.103684 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.103651 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.105264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.105239 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.112364 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.112340 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkvf\" (UniqueName: \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\") pod \"maas-api-revoke-sub-3d4156f1-z97qd\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.201858 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.201765 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:15.480890 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:15.480861 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd"] Oct 30 19:40:15.482875 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:40:15.482845 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df2634e_1030_4ff9_bc51_2e6884f73513.slice/crio-71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582 WatchSource:0}: Error finding container 71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582: Status 404 returned error can't find the container with id 71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582 Oct 30 19:40:16.254870 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:16.254828 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" event={"ID":"0df2634e-1030-4ff9-bc51-2e6884f73513","Type":"ContainerStarted","Data":"6b3bfb739efba48be67db3a268e8bd1a3ad71d14e2f8a4312656deaecec84857"} Oct 30 19:40:16.255213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:16.254896 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" event={"ID":"0df2634e-1030-4ff9-bc51-2e6884f73513","Type":"ContainerStarted","Data":"71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582"} Oct 30 19:40:16.274840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:16.274790 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" podStartSLOduration=2.274774594 podStartE2EDuration="2.274774594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:16.272799888 +0000 UTC m=+1222.249025844" watchObservedRunningTime="2026-10-30 19:40:16.274774594 +0000 UTC m=+1222.251000534" Oct 30 19:40:17.263058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:17.263024 6907 generic.go:356] "Generic (PLEG): container finished" podID="0df2634e-1030-4ff9-bc51-2e6884f73513" containerID="6b3bfb739efba48be67db3a268e8bd1a3ad71d14e2f8a4312656deaecec84857" exitCode=0 Oct 30 19:40:17.263492 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:17.263085 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" event={"ID":"0df2634e-1030-4ff9-bc51-2e6884f73513","Type":"ContainerDied","Data":"6b3bfb739efba48be67db3a268e8bd1a3ad71d14e2f8a4312656deaecec84857"} Oct 30 19:40:18.411576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.411550 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:18.436340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.436313 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\" (UniqueName: \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\") pod \"0df2634e-1030-4ff9-bc51-2e6884f73513\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " Oct 30 19:40:18.436545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.436377 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\") pod \"0df2634e-1030-4ff9-bc51-2e6884f73513\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " Oct 30 19:40:18.436545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.436434 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\") pod \"0df2634e-1030-4ff9-bc51-2e6884f73513\" (UID: \"0df2634e-1030-4ff9-bc51-2e6884f73513\") " Oct 30 19:40:18.436782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.436754 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca" pod "0df2634e-1030-4ff9-bc51-2e6884f73513" (UID: "0df2634e-1030-4ff9-bc51-2e6884f73513"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:18.438548 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.438520 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca" pod "0df2634e-1030-4ff9-bc51-2e6884f73513" (UID: "0df2634e-1030-4ff9-bc51-2e6884f73513"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:18.438853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.438828 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf" pod "0df2634e-1030-4ff9-bc51-2e6884f73513" (UID: "0df2634e-1030-4ff9-bc51-2e6884f73513"). InnerVolumeSpecName "kube-api-access-wtkvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:18.537365 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.537331 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkvf\" (UniqueName: \"kubernetes.io/projected/0df2634e-1030-4ff9-bc51-2e6884f73513-kube-api-access-wtkvf\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.537365 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.537364 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.537542 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.537374 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df2634e-1030-4ff9-bc51-2e6884f73513-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.762714 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.762684 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:40:18.840371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840274 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840371 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840326 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840399 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840428 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840575 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840644 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840622 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840658 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840695 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840727 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:18.840856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840824 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\") pod \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\" (UID: \"ceac5dfc-dc74-4b6e-aa81-e025867e4920\") " Oct 30 19:40:18.840856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840836 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:18.841115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.840912 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:18.841115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841009 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:18.841115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841023 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:18.841339 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841315 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.841409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841347 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.841409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841363 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.841409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841376 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.841409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.841391 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.842957 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.842930 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "kube-api-access-4fwgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:18.843061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.842953 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:18.843061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.842960 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:18.843061 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.842974 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token" pod "ceac5dfc-dc74-4b6e-aa81-e025867e4920" (UID: "ceac5dfc-dc74-4b6e-aa81-e025867e4920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:18.941940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.941901 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwgj\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-kube-api-access-4fwgj\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.941940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.941933 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.941940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.941944 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:18.941940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:18.941952 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceac5dfc-dc74-4b6e-aa81-e025867e4920-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:19.277228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.277188 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" Oct 30 19:40:19.277228 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.277211 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd" event={"ID":"0df2634e-1030-4ff9-bc51-2e6884f73513","Type":"ContainerDied","Data":"71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582"} Oct 30 19:40:19.277496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.277271 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71dd91e7b3210dd2d2a99c05fea860c3e211bee29654510899663413c0e74582" Oct 30 19:40:19.279128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.279098 6907 generic.go:356] "Generic (PLEG): container finished" podID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerID="65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0" exitCode=0 Oct 30 19:40:19.279249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.279149 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerDied","Data":"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0"} Oct 30 19:40:19.279249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.279164 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" Oct 30 19:40:19.279249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.279189 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2" event={"ID":"ceac5dfc-dc74-4b6e-aa81-e025867e4920","Type":"ContainerDied","Data":"951e956f90afbd817da25f51900de43072d4bd5fcd62cb943e28beff6009403e"} Oct 30 19:40:19.279249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.279220 6907 scope.go:122] "RemoveContainer" containerID="65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0" Oct 30 19:40:19.294592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.294568 6907 scope.go:122] "RemoveContainer" containerID="4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e" Oct 30 19:40:19.304889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.304868 6907 scope.go:122] "RemoveContainer" containerID="65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0" Oct 30 19:40:19.305171 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:40:19.305152 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0\": container with ID starting with 65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0 not found: ID does not exist" containerID="65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0" Oct 30 19:40:19.305217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.305182 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0"} err="failed to get container status \"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0\": rpc error: code = NotFound desc = could not find container \"65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0\": container with ID starting with 65bb2b8ee2f8b54ec5698126ac39062672af1dcb829efe526c67d16afc87ffd0 not found: ID does not exist" Oct 30 19:40:19.305217 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.305201 6907 scope.go:122] "RemoveContainer" containerID="4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e" Oct 30 19:40:19.305468 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:40:19.305434 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e\": container with ID starting with 4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e not found: ID does not exist" containerID="4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e" Oct 30 19:40:19.305600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.305469 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e"} err="failed to get container status \"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e\": rpc error: code = NotFound desc = could not find container \"4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e\": container with ID starting with 4fb3cc244c2e43577db7eef11a8ddf51b38178a47bff2aae9c3289e2b6568f7e not found: ID does not exist" Oct 30 19:40:19.308544 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.308522 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:40:19.318135 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:19.318111 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-88ec2c-openshift-default-5f6bb5db77bjnf2"] Oct 30 19:40:20.150074 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:20.150039 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" path="/var/lib/kubelet/pods/ceac5dfc-dc74-4b6e-aa81-e025867e4920/volumes" Oct 30 19:40:26.258268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:26.258234 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:40:30.026887 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:30.026845 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm"] Oct 30 19:40:30.031782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:30.031750 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-jwqbm"] Oct 30 19:40:30.150037 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:30.149999 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef09ce1-64fc-402f-a531-3794c7c201d6" path="/var/lib/kubelet/pods/0ef09ce1-64fc-402f-a531-3794c7c201d6/volumes" Oct 30 19:40:30.257259 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:30.257226 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:40:37.737004 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:40:37.736955 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:40:41.033456 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:41.033418 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm"] Oct 30 19:40:41.035590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:41.035559 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-45vnm"] Oct 30 19:40:42.149812 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:42.149778 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e77756-b290-461c-82b1-0ca8570eb22a" path="/var/lib/kubelet/pods/52e77756-b290-461c-82b1-0ca8570eb22a/volumes" Oct 30 19:40:56.367472 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:40:56.367427 6907 scope.go:122] "RemoveContainer" containerID="64f58ee84c898e29b22aaf12deb2f71f08d5ab167b1d1bd1b691689782bacdaa" Oct 30 19:41:05.027137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:05.027099 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg"] Oct 30 19:41:05.031704 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:05.031675 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vnfvg"] Oct 30 19:41:06.149518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:06.149410 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c02eab-19ea-4646-b0a4-2f6b47240a76" path="/var/lib/kubelet/pods/d1c02eab-19ea-4646-b0a4-2f6b47240a76/volumes" Oct 30 19:41:07.736533 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:41:07.736493 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:41:16.051383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:16.051344 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs"] Oct 30 19:41:16.058397 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:16.058369 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-jrvcs"] Oct 30 19:41:16.149546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:16.149515 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25" path="/var/lib/kubelet/pods/503f1bc1-ca11-4535-9a4e-cdb6cb6b1f25/volumes" Oct 30 19:41:36.047639 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:36.047568 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk"] Oct 30 19:41:36.050285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:36.050257 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-chvpk"] Oct 30 19:41:36.149698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:36.149667 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc00f83-970a-4dda-acf8-3987a7784f94" path="/var/lib/kubelet/pods/3bc00f83-970a-4dda-acf8-3987a7784f94/volumes" Oct 30 19:41:37.736424 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:41:37.736380 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:41:52.866203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:52.866167 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:41:54.029409 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:54.029242 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5"] Oct 30 19:41:54.033486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:54.033423 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-txrs5"] Oct 30 19:41:54.178171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:54.178132 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e294bea-7b40-452e-b272-92c9bea69bc4" path="/var/lib/kubelet/pods/6e294bea-7b40-452e-b272-92c9bea69bc4/volumes" Oct 30 19:41:56.397538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:56.397506 6907 scope.go:122] "RemoveContainer" containerID="6ae099040ee7b433e25e49a42edcc1b31a046d70fee12256a120b464c4419372" Oct 30 19:41:56.407363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:56.407340 6907 scope.go:122] "RemoveContainer" containerID="d1a487da3485b23533b28cca0a3092a0a3260ba136794cee74c7a0a6c5c5d70c" Oct 30 19:41:56.416629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:41:56.416607 6907 scope.go:122] "RemoveContainer" containerID="00ed70686c115dd17925237446efed6ce6a05783cf9010e1a872881eb54d7106" Oct 30 19:42:06.967434 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:06.967392 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:42:07.737488 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:42:07.737430 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:42:11.679470 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:11.664437 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:42:12.065572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:12.065496 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh"] Oct 30 19:42:12.087224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:12.087172 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5fqsh"] Oct 30 19:42:12.158772 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:12.158733 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb792789-4752-4353-8c11-180dcecdf3f3" path="/var/lib/kubelet/pods/cb792789-4752-4353-8c11-180dcecdf3f3/volumes" Oct 30 19:42:26.050014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:26.049978 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:42:27.199622 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.199584 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:42:27.200284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200262 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:27.200383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200285 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:27.200383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200300 6907 cpu_manager.go:414] "containerMap: removing container" podUID="0df2634e-1030-4ff9-bc51-2e6884f73513" containerName="revoke-keys" Oct 30 19:42:27.200383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200311 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df2634e-1030-4ff9-bc51-2e6884f73513" containerName="revoke-keys" Oct 30 19:42:27.200383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200363 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:27.200383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200371 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:27.200664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200510 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:27.200664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.200532 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df2634e-1030-4ff9-bc51-2e6884f73513" containerName="revoke-keys" Oct 30 19:42:27.203844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.203820 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.219411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.219387 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ad6b27-openshift-default-dockercfg-nfvjq\"" Oct 30 19:42:27.219767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.219744 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333006 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333083 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333146 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333191 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333227 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333258 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.333565 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333295 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.334621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333631 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.334621 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.333676 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lngj\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.434725 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.434682 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.434935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.434777 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.434935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.434891 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.434935 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.434927 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435001 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435033 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435081 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435120 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435108 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lngj\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435160 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435325 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435197 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435395 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.435739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435712 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.436006 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.435972 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.436194 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.436169 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.438712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.438670 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.444943 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.444870 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.445763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.445714 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lngj\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.459243 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.459169 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\") pod \"e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.523590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.523547 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:27.806177 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.806150 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:42:27.808773 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:42:27.808747 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63cf008_425d_4cca_83b6_9a4cda401c24.slice/crio-9b883f4a5c6b397d063277ed9733d0019e0799088eae2dd7a6df22808f7e94fd WatchSource:0}: Error finding container 9b883f4a5c6b397d063277ed9733d0019e0799088eae2dd7a6df22808f7e94fd: Status 404 returned error can't find the container with id 9b883f4a5c6b397d063277ed9733d0019e0799088eae2dd7a6df22808f7e94fd Oct 30 19:42:27.811169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.811128 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:27.811278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.811227 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:27.811321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:27.811273 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:28.163594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:28.163499 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerStarted","Data":"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b"} Oct 30 19:42:28.163594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:28.163560 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerStarted","Data":"9b883f4a5c6b397d063277ed9733d0019e0799088eae2dd7a6df22808f7e94fd"} Oct 30 19:42:28.193214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:28.193151 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podStartSLOduration=1.19313619 podStartE2EDuration="1.19313619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:28.19055505 +0000 UTC m=+1354.166780994" watchObservedRunningTime="2026-10-30 19:42:28.19313619 +0000 UTC m=+1354.169362134" Oct 30 19:42:28.523930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:28.523889 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:28.529084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:28.529056 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:29.169034 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:29.169004 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:29.170099 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:29.170077 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:30.031952 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:30.031916 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf"] Oct 30 19:42:30.035235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:30.035204 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nzwkf"] Oct 30 19:42:30.150454 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:30.150421 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8deb8b3e-16e4-4acb-bf14-5b785f2763ec" path="/var/lib/kubelet/pods/8deb8b3e-16e4-4acb-bf14-5b785f2763ec/volumes" Oct 30 19:42:31.769205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.769161 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:42:31.770024 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.770004 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceac5dfc-dc74-4b6e-aa81-e025867e4920" containerName="istio-proxy" Oct 30 19:42:31.773652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.773631 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.781054 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.781032 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ad6b27\"" Oct 30 19:42:31.806913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.806872 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:42:31.868417 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.868385 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0c8f07cb-dockercfg-tqhvc\"" Oct 30 19:42:31.882126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.882086 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzm7\" (UniqueName: \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.882326 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.882207 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.982999 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.982963 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzm7\" (UniqueName: \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.983208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.983068 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.983823 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.983802 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:31.992520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:31.992489 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzm7\" (UniqueName: \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\") pod \"payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:32.088680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:32.088581 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:32.377961 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:32.377937 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:42:32.380144 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:42:32.380114 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc6f8c7_b3b5_4a38_8f38_3514f0b830df.slice/crio-8b86b56fb395cd7c01286ebd0b4b12b33863f14f894498b09c3c7bd9f356a1bd WatchSource:0}: Error finding container 8b86b56fb395cd7c01286ebd0b4b12b33863f14f894498b09c3c7bd9f356a1bd: Status 404 returned error can't find the container with id 8b86b56fb395cd7c01286ebd0b4b12b33863f14f894498b09c3c7bd9f356a1bd Oct 30 19:42:33.193994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:33.193961 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" event={"ID":"acc6f8c7-b3b5-4a38-8f38-3514f0b830df","Type":"ContainerStarted","Data":"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8"} Oct 30 19:42:33.194398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:33.194011 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" event={"ID":"acc6f8c7-b3b5-4a38-8f38-3514f0b830df","Type":"ContainerStarted","Data":"8b86b56fb395cd7c01286ebd0b4b12b33863f14f894498b09c3c7bd9f356a1bd"} Oct 30 19:42:33.194398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:33.194151 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:33.216264 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:33.216212 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" podStartSLOduration=2.216197888 podStartE2EDuration="2.216197888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:33.214103645 +0000 UTC m=+1359.190329590" watchObservedRunningTime="2026-10-30 19:42:33.216197888 +0000 UTC m=+1359.192423831" Oct 30 19:42:34.264315 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:34.264280 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:42:37.758182 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:42:37.758087 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:42:44.204229 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:44.204198 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:42:44.394975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:44.394939 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": read tcp 10.134.0.2:52954->10.134.0.142:15021: read: connection reset by peer" start-of-body= Oct 30 19:42:44.395182 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:44.395022 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": read tcp 10.134.0.2:52954->10.134.0.142:15021: read: connection reset by peer" Oct 30 19:42:45.296853 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.296806 6907 generic.go:356] "Generic (PLEG): container finished" podID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerID="ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b" exitCode=0 Oct 30 19:42:45.297245 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.296877 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerDied","Data":"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b"} Oct 30 19:42:45.297289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.297261 6907 scope.go:122] "RemoveContainer" containerID="ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b" Oct 30 19:42:45.298657 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.298622 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:45.298743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.298701 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:45.298743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:45.298736 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:42:46.312199 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:46.312158 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerStarted","Data":"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651"} Oct 30 19:42:46.524700 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:46.524661 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:46.524917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:46.524895 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:42:46.524997 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:46.524976 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:42:47.524562 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:47.524519 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:42:47.524998 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:47.524595 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:42:48.524546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:48.524507 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:42:48.524743 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:48.524577 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:42:49.524206 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:49.524169 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:42:49.524406 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:49.524235 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:42:50.524534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:50.524493 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 19:42:50.524962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:50.524554 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 19:42:50.781471 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:50.781370 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:42:51.958359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:51.958321 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:51.958981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:51.958959 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:51.960049 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:51.959979 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:42:56.454131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:56.454099 6907 scope.go:122] "RemoveContainer" containerID="c7ac16082d26049caf37a476dd90cd4aebc128de677df8d4f7c7677ad9dbf0c8" Oct 30 19:42:56.463866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:56.463845 6907 scope.go:122] "RemoveContainer" containerID="580a833e78a57283de937c7846075c8ef214fe22d61f330b087581691df87650" Oct 30 19:42:56.472561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:56.472540 6907 scope.go:122] "RemoveContainer" containerID="a0391f7f32ed6d784a2c3e74fcff596c22314a7a2cf11c6f6ddf44f43860b7f6" Oct 30 19:42:56.480877 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:42:56.480855 6907 scope.go:122] "RemoveContainer" containerID="e92a0e7aaa9c16aadde5effe2e36d1e47348da60e93fd9051ec023142cbc118f" Oct 30 19:43:06.334195 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:06.334158 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:43:07.736968 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:43:07.736924 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:43:17.689872 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.689832 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8"] Oct 30 19:43:17.693907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.693883 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.702937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.702899 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:43:17.703076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.702967 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:43:17.703076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.703042 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:43:17.703175 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.703118 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:43:17.706050 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.706022 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8"] Oct 30 19:43:17.750163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.750130 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v592c\" (UniqueName: \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.750348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.750173 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.750348 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.750289 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.851115 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.851076 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v592c\" (UniqueName: \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.851321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.851124 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.851321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.851180 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.851950 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.851914 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.853589 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.853570 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:17.860600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:17.860578 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v592c\" (UniqueName: \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\") pod \"maas-api-revoke-sub-88dcd18b-rqvp8\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:18.010040 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:18.010001 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:18.286418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:18.286393 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8"] Oct 30 19:43:18.288681 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:43:18.288651 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0718fdc1_399e_477c_be1f_5af3e2de4b88.slice/crio-ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9 WatchSource:0}: Error finding container ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9: Status 404 returned error can't find the container with id ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9 Oct 30 19:43:18.536017 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:18.535940 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" event={"ID":"0718fdc1-399e-477c-be1f-5af3e2de4b88","Type":"ContainerStarted","Data":"ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9"} Oct 30 19:43:19.543041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:19.543002 6907 generic.go:356] "Generic (PLEG): container finished" podID="0718fdc1-399e-477c-be1f-5af3e2de4b88" containerID="7bb155e51971f7137a52199a1e4323cae5eadc7e39dc30ab36cd4a864b503a26" exitCode=0 Oct 30 19:43:19.543475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:19.543078 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" event={"ID":"0718fdc1-399e-477c-be1f-5af3e2de4b88","Type":"ContainerDied","Data":"7bb155e51971f7137a52199a1e4323cae5eadc7e39dc30ab36cd4a864b503a26"} Oct 30 19:43:20.691398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.691368 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:20.780490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.780425 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\" (UniqueName: \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\") pod \"0718fdc1-399e-477c-be1f-5af3e2de4b88\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " Oct 30 19:43:20.780695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.780565 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\") pod \"0718fdc1-399e-477c-be1f-5af3e2de4b88\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " Oct 30 19:43:20.780695 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.780636 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\") pod \"0718fdc1-399e-477c-be1f-5af3e2de4b88\" (UID: \"0718fdc1-399e-477c-be1f-5af3e2de4b88\") " Oct 30 19:43:20.781022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.780999 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca" pod "0718fdc1-399e-477c-be1f-5af3e2de4b88" (UID: "0718fdc1-399e-477c-be1f-5af3e2de4b88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:20.782551 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.782524 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c" pod "0718fdc1-399e-477c-be1f-5af3e2de4b88" (UID: "0718fdc1-399e-477c-be1f-5af3e2de4b88"). InnerVolumeSpecName "kube-api-access-v592c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:20.782742 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.782722 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca" pod "0718fdc1-399e-477c-be1f-5af3e2de4b88" (UID: "0718fdc1-399e-477c-be1f-5af3e2de4b88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:20.882305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.882216 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:20.882305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.882248 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0718fdc1-399e-477c-be1f-5af3e2de4b88-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:20.882305 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:20.882259 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v592c\" (UniqueName: \"kubernetes.io/projected/0718fdc1-399e-477c-be1f-5af3e2de4b88-kube-api-access-v592c\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:21.024689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:21.024650 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v"] Oct 30 19:43:21.027907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:21.027878 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-7pf5v"] Oct 30 19:43:21.556947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:21.556915 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" Oct 30 19:43:21.556947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:21.556938 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8" event={"ID":"0718fdc1-399e-477c-be1f-5af3e2de4b88","Type":"ContainerDied","Data":"ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9"} Oct 30 19:43:21.557193 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:21.556993 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca09bddd4bca3ad288cbf09405178e8fb62fd761357ccd1bcca73eaf7636c8d9" Oct 30 19:43:22.149875 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:22.149839 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17b24381-6516-49a4-aa32-835be3d90b13" path="/var/lib/kubelet/pods/17b24381-6516-49a4-aa32-835be3d90b13/volumes" Oct 30 19:43:28.847983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.847933 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:43:28.876755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.876723 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt"] Oct 30 19:43:28.877212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.877199 6907 cpu_manager.go:414] "containerMap: removing container" podUID="0718fdc1-399e-477c-be1f-5af3e2de4b88" containerName="revoke-keys" Oct 30 19:43:28.877263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.877214 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0718fdc1-399e-477c-be1f-5af3e2de4b88" containerName="revoke-keys" Oct 30 19:43:28.877362 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.877350 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="0718fdc1-399e-477c-be1f-5af3e2de4b88" containerName="revoke-keys" Oct 30 19:43:28.880875 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.880855 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:28.889230 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.889159 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:43:28.889837 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.889806 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:43:28.889954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.889893 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:43:28.889954 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.889900 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:43:28.891726 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.891703 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt"] Oct 30 19:43:28.987631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.987596 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:28.987835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.987663 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbfw\" (UniqueName: \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:28.987835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:28.987787 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.093584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.089068 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.094112 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.092172 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.094296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.094276 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbfw\" (UniqueName: \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.094860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.094837 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.098068 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.097981 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.105152 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.105124 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbfw\" (UniqueName: \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.196130 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.196089 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:29.477126 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.477082 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt"] Oct 30 19:43:29.479766 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:43:29.479730 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48ddb2c_4382_4158_830b_ac5f36b7f19c.slice/crio-58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6 WatchSource:0}: Error finding container 58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6: Status 404 returned error can't find the container with id 58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6 Oct 30 19:43:29.606087 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:29.606055 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" event={"ID":"d48ddb2c-4382-4158-830b-ac5f36b7f19c","Type":"ContainerStarted","Data":"58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6"} Oct 30 19:43:30.613407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:30.613310 6907 generic.go:356] "Generic (PLEG): container finished" podID="d48ddb2c-4382-4158-830b-ac5f36b7f19c" containerID="bf20df480c2baa2894fb4331b40dfe4505d5a4f625cb6826520b9e21df545311" exitCode=0 Oct 30 19:43:30.613821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:30.613402 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" event={"ID":"d48ddb2c-4382-4158-830b-ac5f36b7f19c","Type":"ContainerDied","Data":"bf20df480c2baa2894fb4331b40dfe4505d5a4f625cb6826520b9e21df545311"} Oct 30 19:43:31.759756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.759722 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:31.926043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.925960 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\") pod \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " Oct 30 19:43:31.926204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.926121 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\") pod \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " Oct 30 19:43:31.926204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.926162 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\" (UniqueName: \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\") pod \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\" (UID: \"d48ddb2c-4382-4158-830b-ac5f36b7f19c\") " Oct 30 19:43:31.926318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.926291 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca" pod "d48ddb2c-4382-4158-830b-ac5f36b7f19c" (UID: "d48ddb2c-4382-4158-830b-ac5f36b7f19c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:31.926620 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.926601 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:31.928186 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.928155 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca" pod "d48ddb2c-4382-4158-830b-ac5f36b7f19c" (UID: "d48ddb2c-4382-4158-830b-ac5f36b7f19c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:43:31.928293 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:31.928272 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw" pod "d48ddb2c-4382-4158-830b-ac5f36b7f19c" (UID: "d48ddb2c-4382-4158-830b-ac5f36b7f19c"). InnerVolumeSpecName "kube-api-access-ltbfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:32.028226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.028180 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48ddb2c-4382-4158-830b-ac5f36b7f19c-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:32.028226 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.028212 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbfw\" (UniqueName: \"kubernetes.io/projected/d48ddb2c-4382-4158-830b-ac5f36b7f19c-kube-api-access-ltbfw\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:32.040572 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.040536 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv"] Oct 30 19:43:32.045095 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.045063 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-48tqv"] Oct 30 19:43:32.149805 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.149773 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8660d443-c074-4d6b-977e-e4a6fbeac905" path="/var/lib/kubelet/pods/8660d443-c074-4d6b-977e-e4a6fbeac905/volumes" Oct 30 19:43:32.626651 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.626623 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" Oct 30 19:43:32.626831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.626659 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt" event={"ID":"d48ddb2c-4382-4158-830b-ac5f36b7f19c","Type":"ContainerDied","Data":"58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6"} Oct 30 19:43:32.626831 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:32.626706 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c1e3a39379502c3ac178ed33a2c6e17abfddb24428e6f754d4d86f277bd7f6" Oct 30 19:43:37.735797 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:43:37.735751 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:43:39.048678 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:39.048639 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:43:40.795986 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:40.795940 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt"] Oct 30 19:43:40.798131 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:40.798104 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ad6b27-4w9zt"] Oct 30 19:43:42.150083 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:42.150046 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48ddb2c-4382-4158-830b-ac5f36b7f19c" path="/var/lib/kubelet/pods/d48ddb2c-4382-4158-830b-ac5f36b7f19c/volumes" Oct 30 19:43:42.423376 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:42.423290 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:43:44.049346 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:44.049293 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" containerID="cri-o://7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8" gracePeriod=25 Oct 30 19:43:44.202429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:44.202388 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:43:48.799414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:48.799378 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:43:48.799857 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:48.799607 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" containerID="cri-o://98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651" gracePeriod=30 Oct 30 19:43:51.961133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:51.961089 6907 patch_prober.go:29] interesting pod/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:43:51.961629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:51.961154 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:43:54.059038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.059011 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:43:54.171972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.171880 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.171932 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172431 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172294 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172435 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172507 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172548 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172579 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.172904 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172753 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.173241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172985 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.173698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.172708 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.173698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173351 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.173698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173382 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\") pod \"a63cf008-425d-4cca-83b6-9a4cda401c24\" (UID: \"a63cf008-425d-4cca-83b6-9a4cda401c24\") " Oct 30 19:43:54.173698 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173653 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.173974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173777 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:43:54.173974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173850 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.173974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173872 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.173974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173888 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.173974 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.173902 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.175097 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.175069 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:54.175224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.175099 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:43:54.175299 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.175282 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "kube-api-access-7lngj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:43:54.175486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.175460 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo" pod "a63cf008-425d-4cca-83b6-9a4cda401c24" (UID: "a63cf008-425d-4cca-83b6-9a4cda401c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:43:54.176618 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.176593 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:43:54.203073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.203038 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:43:54.274513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.274475 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.274513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.274504 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.274513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.274514 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lngj\" (UniqueName: \"kubernetes.io/projected/a63cf008-425d-4cca-83b6-9a4cda401c24-kube-api-access-7lngj\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.274513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.274523 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf008-425d-4cca-83b6-9a4cda401c24-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.274867 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.274532 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf008-425d-4cca-83b6-9a4cda401c24-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:43:54.782452 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.782407 6907 generic.go:356] "Generic (PLEG): container finished" podID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerID="98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651" exitCode=0 Oct 30 19:43:54.782684 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.782492 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" Oct 30 19:43:54.782684 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.782501 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerDied","Data":"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651"} Oct 30 19:43:54.782684 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.782568 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb" event={"ID":"a63cf008-425d-4cca-83b6-9a4cda401c24","Type":"ContainerDied","Data":"9b883f4a5c6b397d063277ed9733d0019e0799088eae2dd7a6df22808f7e94fd"} Oct 30 19:43:54.782684 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.782601 6907 scope.go:122] "RemoveContainer" containerID="98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651" Oct 30 19:43:54.806512 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.806483 6907 scope.go:122] "RemoveContainer" containerID="ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b" Oct 30 19:43:54.822769 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.822739 6907 scope.go:122] "RemoveContainer" containerID="98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651" Oct 30 19:43:54.823164 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:43:54.823135 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651\": container with ID starting with 98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651 not found: ID does not exist" containerID="98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651" Oct 30 19:43:54.823294 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.823176 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651"} err="failed to get container status \"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651\": rpc error: code = NotFound desc = could not find container \"98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651\": container with ID starting with 98d1d11e4737bfb64480b14885d28a0a46af1b1d4fc867feb9b23d993ed1f651 not found: ID does not exist" Oct 30 19:43:54.823294 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.823205 6907 scope.go:122] "RemoveContainer" containerID="ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b" Oct 30 19:43:54.823838 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:43:54.823580 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b\": container with ID starting with ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b not found: ID does not exist" containerID="ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b" Oct 30 19:43:54.823838 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.823617 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b"} err="failed to get container status \"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b\": rpc error: code = NotFound desc = could not find container \"ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b\": container with ID starting with ce51b7f943918059f466678922b556728e59dd3e2eba57810a5f5d8856fbf70b not found: ID does not exist" Oct 30 19:43:54.827361 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.827332 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:43:54.830754 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:54.830723 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ad6b27-openshift-default-78466fb4b5h5jnb"] Oct 30 19:43:56.181116 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:56.181077 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" path="/var/lib/kubelet/pods/a63cf008-425d-4cca-83b6-9a4cda401c24/volumes" Oct 30 19:43:56.528110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:56.528081 6907 scope.go:122] "RemoveContainer" containerID="cece2f6aa819b1772b08b89cedd264487ce497d8c32285cd4ea5332eb777ed05" Oct 30 19:43:56.570160 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:56.570133 6907 scope.go:122] "RemoveContainer" containerID="2fd5928f820be4ad75a324324c4d423385b24bec1a64980521780bb9e1b79de2" Oct 30 19:43:56.678363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:56.676639 6907 scope.go:122] "RemoveContainer" containerID="a81e046e9f66499ffa465e83a51b417dc7e033209a979bed73f64f45b6ab0ec7" Oct 30 19:43:56.804845 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:43:56.804811 6907 scope.go:122] "RemoveContainer" containerID="585d58089a33f7c073740281f98bf6fb327c5fca310401da41e496e24b88a3d6" Oct 30 19:44:04.201832 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:44:04.201801 6907 logging.go:55] [core] [Channel #1472 SubChannel #1473]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.143:50052", ServerName: "10.134.0.143:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.143:50052: connect: connection refused" Oct 30 19:44:04.306355 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.306288 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:44:04.385737 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.385705 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\" (UniqueName: \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\") pod \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " Oct 30 19:44:04.385955 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.385792 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\") pod \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\" (UID: \"acc6f8c7-b3b5-4a38-8f38-3514f0b830df\") " Oct 30 19:44:04.386180 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.386156 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume" pod "acc6f8c7-b3b5-4a38-8f38-3514f0b830df" (UID: "acc6f8c7-b3b5-4a38-8f38-3514f0b830df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:04.387880 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.387850 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7" pod "acc6f8c7-b3b5-4a38-8f38-3514f0b830df" (UID: "acc6f8c7-b3b5-4a38-8f38-3514f0b830df"). InnerVolumeSpecName "kube-api-access-bfzm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:04.486920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.486877 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzm7\" (UniqueName: \"kubernetes.io/projected/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-kube-api-access-bfzm7\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.486920 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.486912 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acc6f8c7-b3b5-4a38-8f38-3514f0b830df-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:04.887785 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.887743 6907 generic.go:356] "Generic (PLEG): container finished" podID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerID="7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8" exitCode=0 Oct 30 19:44:04.887981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.887797 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" Oct 30 19:44:04.887981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.887808 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" event={"ID":"acc6f8c7-b3b5-4a38-8f38-3514f0b830df","Type":"ContainerDied","Data":"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8"} Oct 30 19:44:04.887981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.887861 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" event={"ID":"acc6f8c7-b3b5-4a38-8f38-3514f0b830df","Type":"ContainerDied","Data":"8b86b56fb395cd7c01286ebd0b4b12b33863f14f894498b09c3c7bd9f356a1bd"} Oct 30 19:44:04.887981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.887893 6907 scope.go:122] "RemoveContainer" containerID="7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8" Oct 30 19:44:04.897864 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.897842 6907 scope.go:122] "RemoveContainer" containerID="7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8" Oct 30 19:44:04.898129 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:44:04.898106 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8\": container with ID starting with 7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8 not found: ID does not exist" containerID="7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8" Oct 30 19:44:04.898204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.898141 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8"} err="failed to get container status \"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8\": rpc error: code = NotFound desc = could not find container \"7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8\": container with ID starting with 7120a9e1e1dd20b104f96866c5b5071e3a99802244fc4a149d34952b73d0c5d8 not found: ID does not exist" Oct 30 19:44:04.910677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.910641 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:44:04.916063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:04.916038 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4"] Oct 30 19:44:05.201840 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:05.201742 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ad6b27-865ccccsj4" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.143:50052\" within 1s: context deadline exceeded" Oct 30 19:44:06.149946 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:06.149843 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" path="/var/lib/kubelet/pods/acc6f8c7-b3b5-4a38-8f38-3514f0b830df/volumes" Oct 30 19:44:07.737940 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:44:07.737893 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:44:13.433141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:13.433099 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:44:16.389587 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:16.389533 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:16.390249 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:16.390214 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:16.640531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:16.638569 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-vzsw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:16.640531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:16.638647 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-vzsw7" podUID="0818d712-9d8b-4db5-9b05-ffca59cfdea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:26.780555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.780515 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl"] Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781003 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781017 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781033 6907 cpu_manager.go:414] "containerMap: removing container" podUID="d48ddb2c-4382-4158-830b-ac5f36b7f19c" containerName="revoke-keys" Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781039 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48ddb2c-4382-4158-830b-ac5f36b7f19c" containerName="revoke-keys" Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781052 6907 cpu_manager.go:414] "containerMap: removing container" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:26.781073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781057 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:26.781320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781080 6907 cpu_manager.go:414] "containerMap: removing container" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" Oct 30 19:44:26.781320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781087 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" Oct 30 19:44:26.781320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781170 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:26.781320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781180 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48ddb2c-4382-4158-830b-ac5f36b7f19c" containerName="revoke-keys" Oct 30 19:44:26.781320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.781199 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc6f8c7-b3b5-4a38-8f38-3514f0b830df" containerName="payload-pre-processing" Oct 30 19:44:26.784601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.784583 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:26.793988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.793961 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl"] Oct 30 19:44:26.881732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.881695 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:26.882003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.881858 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:44:26.882003 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.881978 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:44:26.882139 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.882022 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:44:26.976712 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.976678 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:26.976895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.976792 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlpx\" (UniqueName: \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:26.976895 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:26.976862 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.078482 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.078357 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.078658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.078516 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlpx\" (UniqueName: \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.078658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.078607 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.079262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.079242 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.080807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.080782 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.091554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.091533 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlpx\" (UniqueName: \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\") pod \"maas-api-revoke-sub-452b6d7b-gf2xl\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.099418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.099395 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:27.383996 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.383927 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl"] Oct 30 19:44:27.386547 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:44:27.386518 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad4c916_4abf_46e1_b600_62c8d622dfde.slice/crio-cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec WatchSource:0}: Error finding container cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec: Status 404 returned error can't find the container with id cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec Oct 30 19:44:27.466927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:27.466888 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:44:28.064830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:28.064786 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" event={"ID":"dad4c916-4abf-46e1-b600-62c8d622dfde","Type":"ContainerStarted","Data":"c35970b648b58482a1a2aaac1d7a6d2a801a18754c92a9de6beb39cd136861dc"} Oct 30 19:44:28.065378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:28.064861 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" event={"ID":"dad4c916-4abf-46e1-b600-62c8d622dfde","Type":"ContainerStarted","Data":"cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec"} Oct 30 19:44:28.091976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:28.091902 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" podStartSLOduration=2.0918818 podStartE2EDuration="2.0918818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:28.089005933 +0000 UTC m=+1474.065231879" watchObservedRunningTime="2026-10-30 19:44:28.0918818 +0000 UTC m=+1474.068107747" Oct 30 19:44:29.114384 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:29.107984 6907 generic.go:356] "Generic (PLEG): container finished" podID="dad4c916-4abf-46e1-b600-62c8d622dfde" containerID="c35970b648b58482a1a2aaac1d7a6d2a801a18754c92a9de6beb39cd136861dc" exitCode=0 Oct 30 19:44:29.114384 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:29.108036 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" event={"ID":"dad4c916-4abf-46e1-b600-62c8d622dfde","Type":"ContainerDied","Data":"c35970b648b58482a1a2aaac1d7a6d2a801a18754c92a9de6beb39cd136861dc"} Oct 30 19:44:31.033910 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.033869 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:31.226423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.225956 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-rp9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:31.226423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.226014 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-rp9rd" podUID="147200da-9ad7-4a69-b764-e074edb12c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:31.248487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.234164 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\") pod \"dad4c916-4abf-46e1-b600-62c8d622dfde\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " Oct 30 19:44:31.248487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.234312 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\" (UniqueName: \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\") pod \"dad4c916-4abf-46e1-b600-62c8d622dfde\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " Oct 30 19:44:31.248487 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.234375 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\") pod \"dad4c916-4abf-46e1-b600-62c8d622dfde\" (UID: \"dad4c916-4abf-46e1-b600-62c8d622dfde\") " Oct 30 19:44:31.251224 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.250701 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca" pod "dad4c916-4abf-46e1-b600-62c8d622dfde" (UID: "dad4c916-4abf-46e1-b600-62c8d622dfde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:31.252611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.252290 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" event={"ID":"dad4c916-4abf-46e1-b600-62c8d622dfde","Type":"ContainerDied","Data":"cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec"} Oct 30 19:44:31.252611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.252352 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5021f0e45fa6a3119ce362dea3f429d26ff88141a94efb5b789fe47b1557ec" Oct 30 19:44:31.252611 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.252433 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl" Oct 30 19:44:31.282689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.280643 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca" pod "dad4c916-4abf-46e1-b600-62c8d622dfde" (UID: "dad4c916-4abf-46e1-b600-62c8d622dfde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:31.282689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.281232 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx" pod "dad4c916-4abf-46e1-b600-62c8d622dfde" (UID: "dad4c916-4abf-46e1-b600-62c8d622dfde"). InnerVolumeSpecName "kube-api-access-jdlpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:31.335992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.335932 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlpx\" (UniqueName: \"kubernetes.io/projected/dad4c916-4abf-46e1-b600-62c8d622dfde-kube-api-access-jdlpx\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:31.335992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.335991 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:31.336287 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.336009 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad4c916-4abf-46e1-b600-62c8d622dfde-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:31.390680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.390629 6907 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75bf867b6c-pns2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:31.390901 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:31.390699 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75bf867b6c-pns2g" podUID="3a9a5b0e-97f2-495a-89e7-516b96f2887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:37.737383 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:44:37.737341 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:44:38.692129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:38.692098 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:44:42.500322 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.500281 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n"] Oct 30 19:44:42.500899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.500880 6907 cpu_manager.go:414] "containerMap: removing container" podUID="dad4c916-4abf-46e1-b600-62c8d622dfde" containerName="revoke-keys" Oct 30 19:44:42.500975 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.500900 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad4c916-4abf-46e1-b600-62c8d622dfde" containerName="revoke-keys" Oct 30 19:44:42.501021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.500996 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad4c916-4abf-46e1-b600-62c8d622dfde" containerName="revoke-keys" Oct 30 19:44:42.501021 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.501005 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cf008-425d-4cca-83b6-9a4cda401c24" containerName="istio-proxy" Oct 30 19:44:42.504582 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.504560 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.513212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.513191 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:42.513789 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.513775 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:44:42.513896 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.513879 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:44:42.519012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.518977 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n"] Oct 30 19:44:42.612802 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.612774 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:44:42.636663 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.636620 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljq2\" (UniqueName: \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.636923 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.636893 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.637153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.637075 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.738497 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.738418 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.738703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.738543 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljq2\" (UniqueName: \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.738703 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.738586 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.739410 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.739384 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.741759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.741727 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.762539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.762414 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljq2\" (UniqueName: \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\") pod \"maas-api-revoke-sub-3e31a506-ps87n\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:42.819303 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:42.819256 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:43.143594 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:43.143557 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n"] Oct 30 19:44:43.347110 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:43.347018 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" event={"ID":"38e8da9b-783a-4a7a-a507-935e49e66b4f","Type":"ContainerStarted","Data":"9bb3915c9a47458b111d9a56f9b3134a003339cc3d4ad68b6db3fc4dfc21c2ec"} Oct 30 19:44:44.404827 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:44.404771 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" event={"ID":"38e8da9b-783a-4a7a-a507-935e49e66b4f","Type":"ContainerStarted","Data":"74ad7d63797378517cda691f63030199442c4e5e7d60c2774aab5aca3e65915e"} Oct 30 19:44:44.477478 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:44.475436 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" podStartSLOduration=2.47541637 podStartE2EDuration="2.47541637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:44.472017701 +0000 UTC m=+1490.448243649" watchObservedRunningTime="2026-10-30 19:44:44.47541637 +0000 UTC m=+1490.451642316" Oct 30 19:44:44.589560 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:44.572585 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:44:45.416462 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:45.416405 6907 generic.go:356] "Generic (PLEG): container finished" podID="38e8da9b-783a-4a7a-a507-935e49e66b4f" containerID="74ad7d63797378517cda691f63030199442c4e5e7d60c2774aab5aca3e65915e" exitCode=0 Oct 30 19:44:45.416942 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:45.416568 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" event={"ID":"38e8da9b-783a-4a7a-a507-935e49e66b4f","Type":"ContainerDied","Data":"74ad7d63797378517cda691f63030199442c4e5e7d60c2774aab5aca3e65915e"} Oct 30 19:44:46.562784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.562753 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:46.728962 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.728927 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\") pod \"38e8da9b-783a-4a7a-a507-935e49e66b4f\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " Oct 30 19:44:46.729141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.729070 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\") pod \"38e8da9b-783a-4a7a-a507-935e49e66b4f\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " Oct 30 19:44:46.729141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.729114 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\" (UniqueName: \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\") pod \"38e8da9b-783a-4a7a-a507-935e49e66b4f\" (UID: \"38e8da9b-783a-4a7a-a507-935e49e66b4f\") " Oct 30 19:44:46.729393 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.729364 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca" pod "38e8da9b-783a-4a7a-a507-935e49e66b4f" (UID: "38e8da9b-783a-4a7a-a507-935e49e66b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:46.729538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.729499 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.731173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.731147 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca" pod "38e8da9b-783a-4a7a-a507-935e49e66b4f" (UID: "38e8da9b-783a-4a7a-a507-935e49e66b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:46.731262 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.731204 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2" pod "38e8da9b-783a-4a7a-a507-935e49e66b4f" (UID: "38e8da9b-783a-4a7a-a507-935e49e66b4f"). InnerVolumeSpecName "kube-api-access-sljq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:46.830254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.830219 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e8da9b-783a-4a7a-a507-935e49e66b4f-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:46.830254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:46.830254 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljq2\" (UniqueName: \"kubernetes.io/projected/38e8da9b-783a-4a7a-a507-935e49e66b4f-kube-api-access-sljq2\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:47.429806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:47.429773 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" Oct 30 19:44:47.429991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:47.429803 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n" event={"ID":"38e8da9b-783a-4a7a-a507-935e49e66b4f","Type":"ContainerDied","Data":"9bb3915c9a47458b111d9a56f9b3134a003339cc3d4ad68b6db3fc4dfc21c2ec"} Oct 30 19:44:47.429991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:47.429857 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb3915c9a47458b111d9a56f9b3134a003339cc3d4ad68b6db3fc4dfc21c2ec" Oct 30 19:44:49.032340 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:49.032296 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5"] Oct 30 19:44:49.043285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:49.043253 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd80ce20-tscj5"] Oct 30 19:44:50.150124 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:50.150088 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a98dd22-4dd2-4b8b-8728-b88323a5a1f4" path="/var/lib/kubelet/pods/0a98dd22-4dd2-4b8b-8728-b88323a5a1f4/volumes" Oct 30 19:44:53.337856 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:53.337820 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:44:54.591272 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:54.591236 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:44:54.713132 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:54.713097 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:44:54.823500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:54.823470 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:44:54.873499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:54.873394 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:44:57.128706 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:57.128668 6907 scope.go:122] "RemoveContainer" containerID="61dad6a82398ebb1fbef7d42691b6516acd7302ec5dbaf142bf6f508c1c1e4ba" Oct 30 19:44:57.210357 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:57.210331 6907 scope.go:122] "RemoveContainer" containerID="c5c897e38227dc0db5feceeab150bd01d2c31c518fec8b6bbc2a70046ed0d628" Oct 30 19:44:58.010042 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:44:58.010005 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:45:02.977680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.977641 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7"] Oct 30 19:45:02.978128 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.978115 6907 cpu_manager.go:414] "containerMap: removing container" podUID="38e8da9b-783a-4a7a-a507-935e49e66b4f" containerName="revoke-keys" Oct 30 19:45:02.978173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.978129 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e8da9b-783a-4a7a-a507-935e49e66b4f" containerName="revoke-keys" Oct 30 19:45:02.978241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.978232 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e8da9b-783a-4a7a-a507-935e49e66b4f" containerName="revoke-keys" Oct 30 19:45:02.983261 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.983242 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:02.989331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.989274 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4kw2n\"" Oct 30 19:45:02.989331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.989278 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:02.989331 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.989282 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:02.989645 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.989389 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:02.991765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:02.991088 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7"] Oct 30 19:45:03.090000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.089965 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.090000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.090005 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh7wq\" (UniqueName: \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.090242 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.090039 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.191096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.191059 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.191096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.191096 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh7wq\" (UniqueName: \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.191377 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.191135 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.191890 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.191862 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.193545 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.193526 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.205835 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.205803 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh7wq\" (UniqueName: \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\") pod \"maas-api-revoke-sub-233ab402-5xgc7\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.298874 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.298778 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:03.599039 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:03.599012 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7"] Oct 30 19:45:03.601154 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:03.601125 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cc575e_0762_409e_85ec_b70a44467b0a.slice/crio-eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae WatchSource:0}: Error finding container eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae: Status 404 returned error can't find the container with id eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae Oct 30 19:45:04.619625 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:04.619586 6907 generic.go:356] "Generic (PLEG): container finished" podID="b0cc575e-0762-409e-85ec-b70a44467b0a" containerID="849196727cde30ab298c6d073577d718f18a9c3d3a402480789ff374520d80b8" exitCode=0 Oct 30 19:45:04.620041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:04.619676 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" event={"ID":"b0cc575e-0762-409e-85ec-b70a44467b0a","Type":"ContainerDied","Data":"849196727cde30ab298c6d073577d718f18a9c3d3a402480789ff374520d80b8"} Oct 30 19:45:04.620041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:04.619735 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" event={"ID":"b0cc575e-0762-409e-85ec-b70a44467b0a","Type":"ContainerStarted","Data":"eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae"} Oct 30 19:45:05.760664 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.760638 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:05.818114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.818066 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\") pod \"b0cc575e-0762-409e-85ec-b70a44467b0a\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " Oct 30 19:45:05.818314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.818218 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\") pod \"b0cc575e-0762-409e-85ec-b70a44467b0a\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " Oct 30 19:45:05.818314 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.818289 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\" (UniqueName: \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\") pod \"b0cc575e-0762-409e-85ec-b70a44467b0a\" (UID: \"b0cc575e-0762-409e-85ec-b70a44467b0a\") " Oct 30 19:45:05.818676 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.818646 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca" pod "b0cc575e-0762-409e-85ec-b70a44467b0a" (UID: "b0cc575e-0762-409e-85ec-b70a44467b0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:05.820278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.820248 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca" pod "b0cc575e-0762-409e-85ec-b70a44467b0a" (UID: "b0cc575e-0762-409e-85ec-b70a44467b0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:05.820278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.820265 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq" pod "b0cc575e-0762-409e-85ec-b70a44467b0a" (UID: "b0cc575e-0762-409e-85ec-b70a44467b0a"). InnerVolumeSpecName "kube-api-access-jh7wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:05.919154 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.919055 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.919154 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.919100 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh7wq\" (UniqueName: \"kubernetes.io/projected/b0cc575e-0762-409e-85ec-b70a44467b0a-kube-api-access-jh7wq\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:05.919154 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:05.919112 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0cc575e-0762-409e-85ec-b70a44467b0a-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:06.632203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:06.632170 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" Oct 30 19:45:06.632391 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:06.632207 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7" event={"ID":"b0cc575e-0762-409e-85ec-b70a44467b0a","Type":"ContainerDied","Data":"eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae"} Oct 30 19:45:06.632391 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:06.632256 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eab45011a15eb536eb0efb42b552b27bbd2ce20c8a9bf0a4de51768f54f0bcae" Oct 30 19:45:07.735604 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:45:07.735568 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:45:13.838104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:13.838065 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:45:15.221098 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.221054 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:15.221759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.221739 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc575e-0762-409e-85ec-b70a44467b0a" containerName="revoke-keys" Oct 30 19:45:15.221804 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.221761 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc575e-0762-409e-85ec-b70a44467b0a" containerName="revoke-keys" Oct 30 19:45:15.221855 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.221846 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc575e-0762-409e-85ec-b70a44467b0a" containerName="revoke-keys" Oct 30 19:45:15.224939 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.224916 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.236252 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.236214 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:15.238685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.238646 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:15.238825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.238718 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-898sh\"" Oct 30 19:45:15.238825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.238820 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:45:15.238937 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.238917 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 19:45:15.239477 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.239438 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:15.316071 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.316035 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.316234 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.316096 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.316234 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.316208 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.316321 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.316235 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4wx\" (UniqueName: \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.417759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.417722 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.417940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.417789 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.417940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.417835 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.417940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.417865 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4wx\" (UniqueName: \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.418436 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.418410 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.420385 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.420364 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.420571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.420549 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.430375 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.430349 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4wx\" (UniqueName: \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\") pod \"maas-api-788b7ddf75-b98h4\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.541503 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.541415 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:15.832886 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.832810 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:15.836873 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:15.836847 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e82742_eddb_4d91_9f8e_7613b594ab90.slice/crio-b7f031b6430c29867289bbefd4b1e2596c9e7a5b005c6b56a3c890fde558ffbc WatchSource:0}: Error finding container b7f031b6430c29867289bbefd4b1e2596c9e7a5b005c6b56a3c890fde558ffbc: Status 404 returned error can't find the container with id b7f031b6430c29867289bbefd4b1e2596c9e7a5b005c6b56a3c890fde558ffbc Oct 30 19:45:15.838292 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:15.838271 6907 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:45:16.697928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:16.697831 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" event={"ID":"b9e82742-eddb-4d91-9f8e-7613b594ab90","Type":"ContainerStarted","Data":"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056"} Oct 30 19:45:16.697928 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:16.697885 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" event={"ID":"b9e82742-eddb-4d91-9f8e-7613b594ab90","Type":"ContainerStarted","Data":"b7f031b6430c29867289bbefd4b1e2596c9e7a5b005c6b56a3c890fde558ffbc"} Oct 30 19:45:16.728649 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:16.728591 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" podStartSLOduration=1.318727607 podStartE2EDuration="1.728575231s" totalImagesPullingTime="409.847624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:16.727726074 +0000 UTC m=+1522.703952027" watchObservedRunningTime="2026-10-30 19:45:16.728575231 +0000 UTC m=+1522.704801174" Oct 30 19:45:17.704008 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:17.703958 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:19.075475 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:19.075424 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd"] Oct 30 19:45:19.081318 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:19.081288 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z97qd"] Oct 30 19:45:20.149700 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:20.149667 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df2634e-1030-4ff9-bc51-2e6884f73513" path="/var/lib/kubelet/pods/0df2634e-1030-4ff9-bc51-2e6884f73513/volumes" Oct 30 19:45:22.699013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.698978 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:45:22.702989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.702966 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.708807 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.708776 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-77ab1071-openshift-default-dockercfg-lkrsx\"" Oct 30 19:45:22.825457 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.825384 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:45:22.896398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896358 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896413 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896525 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896576 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896559 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhd6l\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896586 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896684 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896706 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896746 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.896841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.896812 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997378 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997336 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997383 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997436 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997496 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997578 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997561 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997609 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997672 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997707 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhd6l\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.997782 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.997748 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.998046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.998020 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.998144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.998083 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.998144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.998129 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.998253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.998222 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.998253 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.998229 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:22.999970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:22.999947 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:23.000133 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.000114 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:23.010931 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.010907 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhd6l\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:23.014357 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.014334 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\") pod \"e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:23.016338 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.016316 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:23.260710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.260610 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:23.260891 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.260823 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" podUID="b9e82742-eddb-4d91-9f8e-7613b594ab90" containerName="maas-api" containerID="cri-o://a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056" gracePeriod=30 Oct 30 19:45:23.266263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.266239 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:23.318730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.318708 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:45:23.344642 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:23.344606 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2af1d72_a7a2_4244_890e_8e2c1253d4e8.slice/crio-7c6a2d97092ecc011a9ffc8e4461789df5ecc6481a3c6ef4ed194815c8017238 WatchSource:0}: Error finding container 7c6a2d97092ecc011a9ffc8e4461789df5ecc6481a3c6ef4ed194815c8017238: Status 404 returned error can't find the container with id 7c6a2d97092ecc011a9ffc8e4461789df5ecc6481a3c6ef4ed194815c8017238 Oct 30 19:45:23.367141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.367107 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:23.367246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.367176 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:23.367246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.367205 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:23.508732 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.508616 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:23.704148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704058 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\") pod \"b9e82742-eddb-4d91-9f8e-7613b594ab90\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " Oct 30 19:45:23.704148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704107 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\" (UniqueName: \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\") pod \"b9e82742-eddb-4d91-9f8e-7613b594ab90\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " Oct 30 19:45:23.704148 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704134 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\") pod \"b9e82742-eddb-4d91-9f8e-7613b594ab90\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " Oct 30 19:45:23.704653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704265 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\") pod \"b9e82742-eddb-4d91-9f8e-7613b594ab90\" (UID: \"b9e82742-eddb-4d91-9f8e-7613b594ab90\") " Oct 30 19:45:23.704653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704437 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca" pod "b9e82742-eddb-4d91-9f8e-7613b594ab90" (UID: "b9e82742-eddb-4d91-9f8e-7613b594ab90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:23.704653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.704640 6907 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e82742-eddb-4d91-9f8e-7613b594ab90-openshift-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.706363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.706334 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls" pod "b9e82742-eddb-4d91-9f8e-7613b594ab90" (UID: "b9e82742-eddb-4d91-9f8e-7613b594ab90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:23.706499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.706386 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs" pod "b9e82742-eddb-4d91-9f8e-7613b594ab90" (UID: "b9e82742-eddb-4d91-9f8e-7613b594ab90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:23.706499 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.706394 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx" pod "b9e82742-eddb-4d91-9f8e-7613b594ab90" (UID: "b9e82742-eddb-4d91-9f8e-7613b594ab90"). InnerVolumeSpecName "kube-api-access-gc4wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:23.741124 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.741089 6907 generic.go:356] "Generic (PLEG): container finished" podID="b9e82742-eddb-4d91-9f8e-7613b594ab90" containerID="a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056" exitCode=0 Oct 30 19:45:23.741313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.741140 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" Oct 30 19:45:23.741313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.741171 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" event={"ID":"b9e82742-eddb-4d91-9f8e-7613b594ab90","Type":"ContainerDied","Data":"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056"} Oct 30 19:45:23.741313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.741242 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4" event={"ID":"b9e82742-eddb-4d91-9f8e-7613b594ab90","Type":"ContainerDied","Data":"b7f031b6430c29867289bbefd4b1e2596c9e7a5b005c6b56a3c890fde558ffbc"} Oct 30 19:45:23.741313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.741273 6907 scope.go:122] "RemoveContainer" containerID="a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056" Oct 30 19:45:23.743038 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.743011 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerStarted","Data":"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8"} Oct 30 19:45:23.743149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.743063 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerStarted","Data":"7c6a2d97092ecc011a9ffc8e4461789df5ecc6481a3c6ef4ed194815c8017238"} Oct 30 19:45:23.751984 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.751952 6907 scope.go:122] "RemoveContainer" containerID="a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056" Oct 30 19:45:23.752235 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:45:23.752218 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056\": container with ID starting with a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056 not found: ID does not exist" containerID="a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056" Oct 30 19:45:23.752286 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.752244 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056"} err="failed to get container status \"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056\": rpc error: code = NotFound desc = could not find container \"a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056\": container with ID starting with a2f290af40c49f66141e96f0cd5653954693ac0aedf3d1ec089ddd908038e056 not found: ID does not exist" Oct 30 19:45:23.783681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.783631 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podStartSLOduration=1.783616892 podStartE2EDuration="1.783616892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:23.781985199 +0000 UTC m=+1529.758211143" watchObservedRunningTime="2026-10-30 19:45:23.783616892 +0000 UTC m=+1529.759842835" Oct 30 19:45:23.803919 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.803888 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:23.805555 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.805525 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-maas-api-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.805692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.805560 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc4wx\" (UniqueName: \"kubernetes.io/projected/b9e82742-eddb-4d91-9f8e-7613b594ab90-kube-api-access-gc4wx\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.805692 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.805581 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9e82742-eddb-4d91-9f8e-7613b594ab90-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:23.812538 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:23.812510 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-788b7ddf75-b98h4"] Oct 30 19:45:24.016843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:24.016813 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:24.021767 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:24.021739 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:24.151637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:24.151600 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e82742-eddb-4d91-9f8e-7613b594ab90" path="/var/lib/kubelet/pods/b9e82742-eddb-4d91-9f8e-7613b594ab90/volumes" Oct 30 19:45:24.751012 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:24.750979 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:24.752125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:24.752109 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:26.859106 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.859078 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:26.859680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.859667 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b9e82742-eddb-4d91-9f8e-7613b594ab90" containerName="maas-api" Oct 30 19:45:26.859733 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.859683 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e82742-eddb-4d91-9f8e-7613b594ab90" containerName="maas-api" Oct 30 19:45:26.859786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.859775 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e82742-eddb-4d91-9f8e-7613b594ab90" containerName="maas-api" Oct 30 19:45:26.864919 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.864899 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:26.879803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.879782 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-77ab1071\"" Oct 30 19:45:26.885214 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.885188 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:26.909981 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.909942 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:26.914629 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.914595 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:26.923076 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.923049 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-77ab1071-dockercfg-rshzj\"" Oct 30 19:45:26.928908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.928880 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:26.944044 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.944003 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:26.944227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.944052 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fn4\" (UniqueName: \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:26.944227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.944156 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn2g\" (UniqueName: \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:26.944227 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:26.944208 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.045675 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.045636 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.045913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.045753 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.045913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.045780 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fn4\" (UniqueName: \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.045913 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.045813 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn2g\" (UniqueName: \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.046455 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.046417 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.046650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.046624 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.061598 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.061574 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fn4\" (UniqueName: \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\") pod \"payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.061720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.061611 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn2g\" (UniqueName: \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\") pod \"payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.146936 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.146855 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:27.150924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.150894 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.169921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.169890 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:45:27.169921 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.169901 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:45:27.170779 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.170758 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-898sh\"" Oct 30 19:45:27.171332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.171307 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-77ab1071-serving-cert\"" Oct 30 19:45:27.171332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.171314 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:45:27.173125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.173101 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:27.178869 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.178845 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.237085 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.236828 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.248580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.248177 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9kv4\" (UniqueName: \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.248580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.248236 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.248580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.248296 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.248580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.248404 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.349114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.349072 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9kv4\" (UniqueName: \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.349283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.349219 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.349386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.349283 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.349468 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.349398 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.350155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.350132 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.352212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.352134 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.352212 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.352177 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.363343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.363319 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9kv4\" (UniqueName: \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\") pod \"maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.462368 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.462343 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:27.464254 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:27.464226 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c49fb8b_f9d9_4d79_881b_79ce15f6c061.slice/crio-2573c9162c02bae1151cdcda342d90aa24c9527769ac83bedbe457efffa2f5be WatchSource:0}: Error finding container 2573c9162c02bae1151cdcda342d90aa24c9527769ac83bedbe457efffa2f5be: Status 404 returned error can't find the container with id 2573c9162c02bae1151cdcda342d90aa24c9527769ac83bedbe457efffa2f5be Oct 30 19:45:27.464356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.464250 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:27.564118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.564069 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:27.568930 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:27.568898 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccadc1e6_e566_4ffe_8671_a1061960fe52.slice/crio-1039dcbea93177c66f2d533b9461b9ad3324bb2c2b4e01da4d2bf5f8c0fd6c50 WatchSource:0}: Error finding container 1039dcbea93177c66f2d533b9461b9ad3324bb2c2b4e01da4d2bf5f8c0fd6c50: Status 404 returned error can't find the container with id 1039dcbea93177c66f2d533b9461b9ad3324bb2c2b4e01da4d2bf5f8c0fd6c50 Oct 30 19:45:27.766763 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.766728 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:27.768815 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:45:27.768787 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e07ca9_ec96_4910_8cd8_5f7dce4b74a1.slice/crio-ee50416e3a74f4c44f9bd75961b90cce585218537e32fe982b828ed76dddc128 WatchSource:0}: Error finding container ee50416e3a74f4c44f9bd75961b90cce585218537e32fe982b828ed76dddc128: Status 404 returned error can't find the container with id ee50416e3a74f4c44f9bd75961b90cce585218537e32fe982b828ed76dddc128 Oct 30 19:45:27.770636 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.770608 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" event={"ID":"ccadc1e6-e566-4ffe-8671-a1061960fe52","Type":"ContainerStarted","Data":"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e"} Oct 30 19:45:27.770806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.770660 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" event={"ID":"ccadc1e6-e566-4ffe-8671-a1061960fe52","Type":"ContainerStarted","Data":"1039dcbea93177c66f2d533b9461b9ad3324bb2c2b4e01da4d2bf5f8c0fd6c50"} Oct 30 19:45:27.770806 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.770783 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:27.772333 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.772312 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" event={"ID":"4c49fb8b-f9d9-4d79-881b-79ce15f6c061","Type":"ContainerStarted","Data":"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a"} Oct 30 19:45:27.772463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.772345 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" event={"ID":"4c49fb8b-f9d9-4d79-881b-79ce15f6c061","Type":"ContainerStarted","Data":"2573c9162c02bae1151cdcda342d90aa24c9527769ac83bedbe457efffa2f5be"} Oct 30 19:45:27.772463 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.772384 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:27.790580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.789779 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" podStartSLOduration=1.789761173 podStartE2EDuration="1.789761173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:27.788204974 +0000 UTC m=+1533.764430919" watchObservedRunningTime="2026-10-30 19:45:27.789761173 +0000 UTC m=+1533.765987125" Oct 30 19:45:27.818254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:27.818163 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" podStartSLOduration=1.818124466 podStartE2EDuration="1.818124466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:27.8076203 +0000 UTC m=+1533.783846245" watchObservedRunningTime="2026-10-30 19:45:27.818124466 +0000 UTC m=+1533.794350412" Oct 30 19:45:28.779833 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:28.779785 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" event={"ID":"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1","Type":"ContainerStarted","Data":"ab6eff2709ca5b36b16434b707418ec38c6d848613df6fd0cb0882eb876ddb61"} Oct 30 19:45:28.780505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:28.779891 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" event={"ID":"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1","Type":"ContainerStarted","Data":"ee50416e3a74f4c44f9bd75961b90cce585218537e32fe982b828ed76dddc128"} Oct 30 19:45:28.780505 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:28.780109 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:28.802676 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:28.802619 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" podStartSLOduration=1.390903917 podStartE2EDuration="1.802603195s" totalImagesPullingTime="411.699278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:28.801276216 +0000 UTC m=+1534.777502163" watchObservedRunningTime="2026-10-30 19:45:28.802603195 +0000 UTC m=+1534.778829138" Oct 30 19:45:29.927059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:29.927020 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:45:34.791617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:34.791588 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:37.737876 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:45:37.737837 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:45:38.783327 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:38.783292 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:38.783860 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:38.783381 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:39.752905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.752870 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:39.753140 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.752932 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:39.853020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.852930 6907 generic.go:356] "Generic (PLEG): container finished" podID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerID="7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8" exitCode=0 Oct 30 19:45:39.853020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.853001 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerDied","Data":"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8"} Oct 30 19:45:39.853480 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.853391 6907 scope.go:122] "RemoveContainer" containerID="7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8" Oct 30 19:45:39.854617 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.854589 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:39.854727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.854658 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:39.854727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:39.854688 6907 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 19:45:40.861423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:40.861390 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerStarted","Data":"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6"} Oct 30 19:45:41.017632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:41.017587 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:41.017841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:41.017809 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:41.017924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:41.017901 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:42.016883 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:42.016854 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:42.017260 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:42.016907 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:43.017079 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:43.017042 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:43.017510 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:43.017100 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:44.017755 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:44.017714 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:44.018205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:44.017775 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:45.017250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:45.017206 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 19:45:45.017439 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:45.017285 6907 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 19:45:46.021164 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:46.021131 6907 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:46.021708 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:46.021572 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:46.022310 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:46.022291 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:45:51.422547 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:51.422507 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:51.441147 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:51.441113 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:52.705184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:52.705154 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:52.705647 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:52.705355 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" podUID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" containerName="maas-api" containerID="cri-o://ab6eff2709ca5b36b16434b707418ec38c6d848613df6fd0cb0882eb876ddb61" gracePeriod=30 Oct 30 19:45:52.961084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:52.960847 6907 generic.go:356] "Generic (PLEG): container finished" podID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" containerID="ab6eff2709ca5b36b16434b707418ec38c6d848613df6fd0cb0882eb876ddb61" exitCode=0 Oct 30 19:45:52.961084 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:52.960905 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" event={"ID":"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1","Type":"ContainerDied","Data":"ab6eff2709ca5b36b16434b707418ec38c6d848613df6fd0cb0882eb876ddb61"} Oct 30 19:45:52.996752 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:52.996718 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:53.085681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.085644 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\") pod \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " Oct 30 19:45:53.085878 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.085754 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\") pod \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " Oct 30 19:45:53.085878 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.085818 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\") pod \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " Oct 30 19:45:53.085991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.085975 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\" (UniqueName: \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\") pod \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\" (UID: \"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1\") " Oct 30 19:45:53.086222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.086187 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca" pod "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" (UID: "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:53.086352 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.086336 6907 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-openshift-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.087987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.087953 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs" pod "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" (UID: "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:53.087987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.087975 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls" pod "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" (UID: "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:53.087987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.087951 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4" pod "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" (UID: "21e07ca9-ec96-4910-8cd8-5f7dce4b74a1"). InnerVolumeSpecName "kube-api-access-j9kv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:53.187502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.187435 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-maas-api-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.187502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.187493 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9kv4\" (UniqueName: \"kubernetes.io/projected/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-kube-api-access-j9kv4\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.187502 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.187504 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:53.968213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.968183 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" Oct 30 19:45:53.968213 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.968199 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh" event={"ID":"21e07ca9-ec96-4910-8cd8-5f7dce4b74a1","Type":"ContainerDied","Data":"ee50416e3a74f4c44f9bd75961b90cce585218537e32fe982b828ed76dddc128"} Oct 30 19:45:53.968798 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.968255 6907 scope.go:122] "RemoveContainer" containerID="ab6eff2709ca5b36b16434b707418ec38c6d848613df6fd0cb0882eb876ddb61" Oct 30 19:45:53.993495 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.993461 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:53.998552 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:53.998520 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-77ab1071-7749d8f477-xmdsh"] Oct 30 19:45:54.150155 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:54.150120 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" path="/var/lib/kubelet/pods/21e07ca9-ec96-4910-8cd8-5f7dce4b74a1/volumes" Oct 30 19:45:54.337561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:54.337529 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:45:56.423167 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.423100 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" podUID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" containerName="payload-processing" containerID="cri-o://88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a" gracePeriod=25 Oct 30 19:45:56.441842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.441804 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" podUID="ccadc1e6-e566-4ffe-8671-a1061960fe52" containerName="payload-pre-processing" containerID="cri-o://584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e" gracePeriod=25 Oct 30 19:45:56.701715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.701690 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:56.738313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.738279 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:56.829289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829250 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\" (UniqueName: \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\") pod \"ccadc1e6-e566-4ffe-8671-a1061960fe52\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " Oct 30 19:45:56.829534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829348 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\") pod \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " Oct 30 19:45:56.829534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829521 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\") pod \"ccadc1e6-e566-4ffe-8671-a1061960fe52\" (UID: \"ccadc1e6-e566-4ffe-8671-a1061960fe52\") " Oct 30 19:45:56.829739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829555 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\" (UniqueName: \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\") pod \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\" (UID: \"4c49fb8b-f9d9-4d79-881b-79ce15f6c061\") " Oct 30 19:45:56.829836 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829808 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume" pod "4c49fb8b-f9d9-4d79-881b-79ce15f6c061" (UID: "4c49fb8b-f9d9-4d79-881b-79ce15f6c061"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:56.829893 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829859 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume" pod "ccadc1e6-e566-4ffe-8671-a1061960fe52" (UID: "ccadc1e6-e566-4ffe-8671-a1061960fe52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:56.829965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829943 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:56.830016 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.829972 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccadc1e6-e566-4ffe-8671-a1061960fe52-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:56.831453 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.831421 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4" pod "ccadc1e6-e566-4ffe-8671-a1061960fe52" (UID: "ccadc1e6-e566-4ffe-8671-a1061960fe52"). InnerVolumeSpecName "kube-api-access-k6fn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:56.831500 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.831461 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g" pod "4c49fb8b-f9d9-4d79-881b-79ce15f6c061" (UID: "4c49fb8b-f9d9-4d79-881b-79ce15f6c061"). InnerVolumeSpecName "kube-api-access-bxn2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:56.931612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.931530 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn2g\" (UniqueName: \"kubernetes.io/projected/4c49fb8b-f9d9-4d79-881b-79ce15f6c061-kube-api-access-bxn2g\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:56.931612 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.931561 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fn4\" (UniqueName: \"kubernetes.io/projected/ccadc1e6-e566-4ffe-8671-a1061960fe52-kube-api-access-k6fn4\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:56.987285 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.987249 6907 generic.go:356] "Generic (PLEG): container finished" podID="ccadc1e6-e566-4ffe-8671-a1061960fe52" containerID="584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e" exitCode=0 Oct 30 19:45:56.987513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.987299 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" Oct 30 19:45:56.987513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.987331 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" event={"ID":"ccadc1e6-e566-4ffe-8671-a1061960fe52","Type":"ContainerDied","Data":"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e"} Oct 30 19:45:56.987513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.987395 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs" event={"ID":"ccadc1e6-e566-4ffe-8671-a1061960fe52","Type":"ContainerDied","Data":"1039dcbea93177c66f2d533b9461b9ad3324bb2c2b4e01da4d2bf5f8c0fd6c50"} Oct 30 19:45:56.987513 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.987428 6907 scope.go:122] "RemoveContainer" containerID="584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e" Oct 30 19:45:56.988847 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.988827 6907 generic.go:356] "Generic (PLEG): container finished" podID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" containerID="88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a" exitCode=0 Oct 30 19:45:56.988941 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.988873 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" Oct 30 19:45:56.988941 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.988903 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" event={"ID":"4c49fb8b-f9d9-4d79-881b-79ce15f6c061","Type":"ContainerDied","Data":"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a"} Oct 30 19:45:56.989042 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.988961 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z" event={"ID":"4c49fb8b-f9d9-4d79-881b-79ce15f6c061","Type":"ContainerDied","Data":"2573c9162c02bae1151cdcda342d90aa24c9527769ac83bedbe457efffa2f5be"} Oct 30 19:45:56.998060 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.998042 6907 scope.go:122] "RemoveContainer" containerID="584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e" Oct 30 19:45:56.998386 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:45:56.998357 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e\": container with ID starting with 584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e not found: ID does not exist" containerID="584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e" Oct 30 19:45:56.998486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.998398 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e"} err="failed to get container status \"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e\": rpc error: code = NotFound desc = could not find container \"584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e\": container with ID starting with 584a2a66f03acaf808e24841dc3c796e3c2251c962093ab0bd7972dec99af03e not found: ID does not exist" Oct 30 19:45:56.998486 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:56.998422 6907 scope.go:122] "RemoveContainer" containerID="88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a" Oct 30 19:45:57.007568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.007552 6907 scope.go:122] "RemoveContainer" containerID="88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a" Oct 30 19:45:57.007803 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:45:57.007788 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a\": container with ID starting with 88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a not found: ID does not exist" containerID="88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a" Oct 30 19:45:57.007854 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.007808 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a"} err="failed to get container status \"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a\": rpc error: code = NotFound desc = could not find container \"88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a\": container with ID starting with 88d3ebde658c164a7643819c954292b0b8cb29555c1ebe491995fc0af85f6b2a not found: ID does not exist" Oct 30 19:45:57.017988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.017954 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:57.020045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.020023 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-77ab1071-9574cbf87-lgncs"] Oct 30 19:45:57.037399 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.037364 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:57.043284 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.043261 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-77ab1071-6d674954cf-c4d8z"] Oct 30 19:45:57.419146 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.419113 6907 scope.go:122] "RemoveContainer" containerID="e3ef25b2e693d75209639f7177650cf6d72822ba0c859062717a5e78ea1dc122" Oct 30 19:45:57.429167 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.429038 6907 scope.go:122] "RemoveContainer" containerID="d6fe72851be0b0f3cd88ba38b5dc624812e9c5830c51f03afdccba47c4745b53" Oct 30 19:45:57.669270 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.669186 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:45:57.669454 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:57.669414 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" containerID="cri-o://c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6" gracePeriod=30 Oct 30 19:45:58.149728 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:58.149687 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" path="/var/lib/kubelet/pods/4c49fb8b-f9d9-4d79-881b-79ce15f6c061/volumes" Oct 30 19:45:58.150059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:45:58.150044 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccadc1e6-e566-4ffe-8671-a1061960fe52" path="/var/lib/kubelet/pods/ccadc1e6-e566-4ffe-8671-a1061960fe52/volumes" Oct 30 19:46:01.023570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:01.023532 6907 patch_prober.go:29] interesting pod/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:46:01.024067 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:01.023601 6907 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:46:02.940032 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.939955 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:46:02.998546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998515 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998546 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998550 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998608 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998639 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998699 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998735 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.998816 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998780 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.999081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998826 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.999081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998879 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:02.999081 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.998886 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\") pod \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\" (UID: \"c2af1d72-a7a2-4244-890e-8e2c1253d4e8\") " Oct 30 19:46:02.999254 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999121 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:02.999600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999354 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:02.999600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999403 6907 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istiod-ca-cert\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.999600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999462 6907 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:02.999600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999478 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:02.999600 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:02.999566 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:03.001129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.001098 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:46:03.001426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.001409 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "kube-api-access-nhd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:03.001559 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.001531 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:03.001710 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.001685 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy" pod "c2af1d72-a7a2-4244-890e-8e2c1253d4e8" (UID: "c2af1d72-a7a2-4244-890e-8e2c1253d4e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:46:03.024689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.024653 6907 generic.go:356] "Generic (PLEG): container finished" podID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerID="c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6" exitCode=0 Oct 30 19:46:03.024830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.024709 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" Oct 30 19:46:03.024830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.024742 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerDied","Data":"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6"} Oct 30 19:46:03.024830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.024804 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm" event={"ID":"c2af1d72-a7a2-4244-890e-8e2c1253d4e8","Type":"ContainerDied","Data":"7c6a2d97092ecc011a9ffc8e4461789df5ecc6481a3c6ef4ed194815c8017238"} Oct 30 19:46:03.024947 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.024833 6907 scope.go:122] "RemoveContainer" containerID="c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6" Oct 30 19:46:03.035207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.035189 6907 scope.go:122] "RemoveContainer" containerID="7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8" Oct 30 19:46:03.044353 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.044335 6907 scope.go:122] "RemoveContainer" containerID="c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6" Oct 30 19:46:03.044649 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:46:03.044627 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6\": container with ID starting with c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6 not found: ID does not exist" containerID="c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6" Oct 30 19:46:03.044705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.044659 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6"} err="failed to get container status \"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6\": rpc error: code = NotFound desc = could not find container \"c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6\": container with ID starting with c497fd844bd19c078a1cf5f2c8b5484438a80889a7b07990e219543e5f4a30a6 not found: ID does not exist" Oct 30 19:46:03.044705 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.044678 6907 scope.go:122] "RemoveContainer" containerID="7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8" Oct 30 19:46:03.044939 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:46:03.044921 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8\": container with ID starting with 7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8 not found: ID does not exist" containerID="7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8" Oct 30 19:46:03.044988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.044945 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8"} err="failed to get container status \"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8\": rpc error: code = NotFound desc = could not find container \"7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8\": container with ID starting with 7d5be99c703ccb9fdc6a2b7aef95cecfc4a416074c37b5c374878c810b3b79b8 not found: ID does not exist" Oct 30 19:46:03.057073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.053154 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:46:03.061702 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.061671 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-77ab1071-openshift-default-647b68f785-8m4qm"] Oct 30 19:46:03.100936 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100907 6907 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-token\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.100936 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100936 6907 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-podinfo\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.101104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100948 6907 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-credential-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.101104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100957 6907 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-workload-socket\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.101104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100966 6907 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-data\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.101104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.100974 6907 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-istio-envoy\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.101104 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.101004 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhd6l\" (UniqueName: \"kubernetes.io/projected/c2af1d72-a7a2-4244-890e-8e2c1253d4e8-kube-api-access-nhd6l\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:03.938730 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.938695 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:46:03.939196 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939182 6907 cpu_manager.go:414] "containerMap: removing container" podUID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" containerName="maas-api" Oct 30 19:46:03.939246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939198 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" containerName="maas-api" Oct 30 19:46:03.939246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939219 6907 cpu_manager.go:414] "containerMap: removing container" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939225 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939234 6907 cpu_manager.go:414] "containerMap: removing container" podUID="ccadc1e6-e566-4ffe-8671-a1061960fe52" containerName="payload-pre-processing" Oct 30 19:46:03.939246 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939239 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccadc1e6-e566-4ffe-8671-a1061960fe52" containerName="payload-pre-processing" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939249 6907 cpu_manager.go:414] "containerMap: removing container" podUID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" containerName="payload-processing" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939254 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" containerName="payload-processing" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939264 6907 cpu_manager.go:414] "containerMap: removing container" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939269 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939338 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939347 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccadc1e6-e566-4ffe-8671-a1061960fe52" containerName="payload-pre-processing" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939355 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c49fb8b-f9d9-4d79-881b-79ce15f6c061" containerName="payload-processing" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939360 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" containerName="istio-proxy" Oct 30 19:46:03.939418 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.939372 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e07ca9-ec96-4910-8cd8-5f7dce4b74a1" containerName="maas-api" Oct 30 19:46:03.944488 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.944463 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:03.957317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.957291 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:46:03.960903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.960826 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:46:03.961059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:03.960909 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:46:04.010251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.010217 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.010251 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.010257 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.010615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.010356 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.010615 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.010488 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczqn\" (UniqueName: \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.023248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.023220 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:46:04.023384 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.023220 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-7ed3da-serving-cert\"" Oct 30 19:46:04.023428 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.023411 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-898sh\"" Oct 30 19:46:04.062765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.062730 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:46:04.066568 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.066551 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.072508 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.072478 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:46:04.076266 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.076245 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.078905 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.078877 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-7ed3da\"" Oct 30 19:46:04.081414 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.081395 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-7ed3da-dockercfg-zpr85\"" Oct 30 19:46:04.088723 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.088700 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:46:04.094257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.094227 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:46:04.111077 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.111049 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.111235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.111084 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.111235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.111109 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.111235 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.111168 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczqn\" (UniqueName: \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.111739 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.111716 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.113641 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.113615 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.113758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.113708 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.128995 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.128967 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczqn\" (UniqueName: \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\") pod \"maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.149356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.149323 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2af1d72-a7a2-4244-890e-8e2c1253d4e8" path="/var/lib/kubelet/pods/c2af1d72-a7a2-4244-890e-8e2c1253d4e8/volumes" Oct 30 19:46:04.220518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.212108 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.220518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.212183 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzgd5\" (UniqueName: \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.220518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.212234 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.220518 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.212263 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8nfp\" (UniqueName: \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.264153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.264112 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:04.313203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.313165 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.313407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.313227 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzgd5\" (UniqueName: \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.313407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.313277 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.313407 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.313304 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8nfp\" (UniqueName: \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.314020 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.313993 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.314118 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.314016 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.325282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.325252 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzgd5\" (UniqueName: \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\") pod \"payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.326149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.326116 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8nfp\" (UniqueName: \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\") pod \"payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.381623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.381586 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:04.394075 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.394036 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:04.567170 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.566971 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:46:04.674591 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:46:04.674563 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c010d_c124_4277_b901_ecddb5492814.slice/crio-31ed56c624088b43db9531b58aebe52e19fdaac2e7e9bc3668464af7a82b20c0 WatchSource:0}: Error finding container 31ed56c624088b43db9531b58aebe52e19fdaac2e7e9bc3668464af7a82b20c0: Status 404 returned error can't find the container with id 31ed56c624088b43db9531b58aebe52e19fdaac2e7e9bc3668464af7a82b20c0 Oct 30 19:46:04.675041 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.675013 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:46:04.705898 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:04.705866 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:46:04.735384 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:46:04.735300 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b04badb_5872_4fdc_aadc_d801fe7db4e1.slice/crio-cff4c5426794a7ae766217638e15222396cb47ad00ac22d7e3afe67b49c9f68e WatchSource:0}: Error finding container cff4c5426794a7ae766217638e15222396cb47ad00ac22d7e3afe67b49c9f68e: Status 404 returned error can't find the container with id cff4c5426794a7ae766217638e15222396cb47ad00ac22d7e3afe67b49c9f68e Oct 30 19:46:05.040670 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.040635 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" event={"ID":"6b04badb-5872-4fdc-aadc-d801fe7db4e1","Type":"ContainerStarted","Data":"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65"} Oct 30 19:46:05.041043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.040701 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" event={"ID":"6b04badb-5872-4fdc-aadc-d801fe7db4e1","Type":"ContainerStarted","Data":"cff4c5426794a7ae766217638e15222396cb47ad00ac22d7e3afe67b49c9f68e"} Oct 30 19:46:05.041043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.040732 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:05.042013 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.041986 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" event={"ID":"b85be873-35a7-4c6f-afd9-e0246f7efb3a","Type":"ContainerStarted","Data":"0395165ed07443d15f1da42392649afb7fc91411f361c1d4cdebb36c45fce8ae"} Oct 30 19:46:05.043423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.043398 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" event={"ID":"845c010d-c124-4277-b901-ecddb5492814","Type":"ContainerStarted","Data":"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec"} Oct 30 19:46:05.043522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.043461 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:05.043522 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.043489 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" event={"ID":"845c010d-c124-4277-b901-ecddb5492814","Type":"ContainerStarted","Data":"31ed56c624088b43db9531b58aebe52e19fdaac2e7e9bc3668464af7a82b20c0"} Oct 30 19:46:05.072571 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.072421 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" podStartSLOduration=1.072406927 podStartE2EDuration="1.072406927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:05.070806539 +0000 UTC m=+1571.047032483" watchObservedRunningTime="2026-10-30 19:46:05.072406927 +0000 UTC m=+1571.048632868" Oct 30 19:46:05.095788 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:05.095737 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" podStartSLOduration=1.095719432 podStartE2EDuration="1.095719432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:05.092867204 +0000 UTC m=+1571.069093149" watchObservedRunningTime="2026-10-30 19:46:05.095719432 +0000 UTC m=+1571.071945433" Oct 30 19:46:06.051677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:06.051637 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" event={"ID":"b85be873-35a7-4c6f-afd9-e0246f7efb3a","Type":"ContainerStarted","Data":"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4"} Oct 30 19:46:06.052119 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:06.051927 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:06.086627 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:06.086573 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" podStartSLOduration=2.663546963 podStartE2EDuration="3.086560021s" totalImagesPullingTime="423.013058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:06.086024368 +0000 UTC m=+1572.062250311" watchObservedRunningTime="2026-10-30 19:46:06.086560021 +0000 UTC m=+1572.062786008" Oct 30 19:46:06.368929 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:06.368851 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:46:07.736536 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:46:07.736497 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:46:12.063388 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:12.063356 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:46:16.055070 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:16.055039 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:46:16.055563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:16.055104 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:46:37.736885 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:46:37.736845 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:46:57.484233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:46:57.484191 6907 scope.go:122] "RemoveContainer" containerID="6b3bfb739efba48be67db3a268e8bd1a3ad71d14e2f8a4312656deaecec84857" Oct 30 19:47:07.738532 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:47:07.738495 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:47:37.737458 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:47:37.737400 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:47:38.152179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.152095 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l"] Oct 30 19:47:38.156018 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.155996 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.162626 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.162599 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kzlqw\"" Oct 30 19:47:38.162890 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.162672 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:38.168283 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.168255 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l"] Oct 30 19:47:38.271674 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.271641 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq58\" (UniqueName: \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.271876 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.271681 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.271990 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.271960 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.372715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.372676 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.372715 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.372726 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq58\" (UniqueName: \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.372999 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.372747 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.373369 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.373346 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.375153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.375128 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.382652 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.382621 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq58\" (UniqueName: \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\") pod \"maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.474819 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.474786 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:38.793494 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:38.789795 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l"] Oct 30 19:47:38.794959 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:47:38.794930 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fb04d4_1902_43f6_bb41_36eeef2156d0.slice/crio-b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69 WatchSource:0}: Error finding container b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69: Status 404 returned error can't find the container with id b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69 Oct 30 19:47:39.662086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:39.662044 6907 generic.go:356] "Generic (PLEG): container finished" podID="37fb04d4-1902-43f6-bb41-36eeef2156d0" containerID="a2eb59d86d5ad8d3116168f75ceae1dc67f3605bdccaeb4679de4b736a01ad28" exitCode=0 Oct 30 19:47:39.662219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:39.662124 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" event={"ID":"37fb04d4-1902-43f6-bb41-36eeef2156d0","Type":"ContainerDied","Data":"a2eb59d86d5ad8d3116168f75ceae1dc67f3605bdccaeb4679de4b736a01ad28"} Oct 30 19:47:39.662219 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:39.662173 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" event={"ID":"37fb04d4-1902-43f6-bb41-36eeef2156d0","Type":"ContainerStarted","Data":"b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69"} Oct 30 19:47:40.813774 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.813748 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:40.900722 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.900683 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\") pod \"37fb04d4-1902-43f6-bb41-36eeef2156d0\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " Oct 30 19:47:40.900903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.900738 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\") pod \"37fb04d4-1902-43f6-bb41-36eeef2156d0\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " Oct 30 19:47:40.900903 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.900881 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\" (UniqueName: \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\") pod \"37fb04d4-1902-43f6-bb41-36eeef2156d0\" (UID: \"37fb04d4-1902-43f6-bb41-36eeef2156d0\") " Oct 30 19:47:40.901144 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.901119 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca" pod "37fb04d4-1902-43f6-bb41-36eeef2156d0" (UID: "37fb04d4-1902-43f6-bb41-36eeef2156d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:40.902924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.902904 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca" pod "37fb04d4-1902-43f6-bb41-36eeef2156d0" (UID: "37fb04d4-1902-43f6-bb41-36eeef2156d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:40.903023 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:40.903010 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58" pod "37fb04d4-1902-43f6-bb41-36eeef2156d0" (UID: "37fb04d4-1902-43f6-bb41-36eeef2156d0"). InnerVolumeSpecName "kube-api-access-5tq58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:41.002225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.002184 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tq58\" (UniqueName: \"kubernetes.io/projected/37fb04d4-1902-43f6-bb41-36eeef2156d0-kube-api-access-5tq58\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:41.002225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.002219 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:41.002225 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.002234 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fb04d4-1902-43f6-bb41-36eeef2156d0-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:41.682632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.682593 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" event={"ID":"37fb04d4-1902-43f6-bb41-36eeef2156d0","Type":"ContainerDied","Data":"b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69"} Oct 30 19:47:41.682632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.682631 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l" Oct 30 19:47:41.682910 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:41.682668 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90ffe5928154e89606196b8a7f06850432f0cf65b14af96149ec29b9406df69" Oct 30 19:47:48.337841 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:48.337792 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:47:48.351658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:48.351620 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:47:53.339729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.339677 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" podUID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" containerName="payload-processing" containerID="cri-o://fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65" gracePeriod=25 Oct 30 19:47:53.352171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.352132 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" podUID="845c010d-c124-4277-b901-ecddb5492814" containerName="payload-pre-processing" containerID="cri-o://19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec" gracePeriod=25 Oct 30 19:47:53.608053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.607990 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:47:53.650142 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.650113 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:47:53.747356 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.747325 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\" (UniqueName: \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\") pod \"845c010d-c124-4277-b901-ecddb5492814\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " Oct 30 19:47:53.747592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.747375 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\" (UniqueName: \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\") pod \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " Oct 30 19:47:53.747592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.747413 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\") pod \"845c010d-c124-4277-b901-ecddb5492814\" (UID: \"845c010d-c124-4277-b901-ecddb5492814\") " Oct 30 19:47:53.747592 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.747584 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\") pod \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\" (UID: \"6b04badb-5872-4fdc-aadc-d801fe7db4e1\") " Oct 30 19:47:53.747987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.747954 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume" pod "845c010d-c124-4277-b901-ecddb5492814" (UID: "845c010d-c124-4277-b901-ecddb5492814"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:53.748098 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.748051 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume" pod "6b04badb-5872-4fdc-aadc-d801fe7db4e1" (UID: "6b04badb-5872-4fdc-aadc-d801fe7db4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:53.749650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.749626 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp" pod "6b04badb-5872-4fdc-aadc-d801fe7db4e1" (UID: "6b04badb-5872-4fdc-aadc-d801fe7db4e1"). InnerVolumeSpecName "kube-api-access-f8nfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:53.749650 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.749636 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5" pod "845c010d-c124-4277-b901-ecddb5492814" (UID: "845c010d-c124-4277-b901-ecddb5492814"). InnerVolumeSpecName "kube-api-access-jzgd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:53.753924 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.753897 6907 generic.go:356] "Generic (PLEG): container finished" podID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" containerID="fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65" exitCode=0 Oct 30 19:47:53.754052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.753948 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" Oct 30 19:47:53.754052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.753973 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" event={"ID":"6b04badb-5872-4fdc-aadc-d801fe7db4e1","Type":"ContainerDied","Data":"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65"} Oct 30 19:47:53.754052 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.754029 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk" event={"ID":"6b04badb-5872-4fdc-aadc-d801fe7db4e1","Type":"ContainerDied","Data":"cff4c5426794a7ae766217638e15222396cb47ad00ac22d7e3afe67b49c9f68e"} Oct 30 19:47:53.754210 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.754059 6907 scope.go:122] "RemoveContainer" containerID="fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65" Oct 30 19:47:53.755666 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.755637 6907 generic.go:356] "Generic (PLEG): container finished" podID="845c010d-c124-4277-b901-ecddb5492814" containerID="19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec" exitCode=0 Oct 30 19:47:53.755800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.755688 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" Oct 30 19:47:53.755800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.755709 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" event={"ID":"845c010d-c124-4277-b901-ecddb5492814","Type":"ContainerDied","Data":"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec"} Oct 30 19:47:53.755800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.755742 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v" event={"ID":"845c010d-c124-4277-b901-ecddb5492814","Type":"ContainerDied","Data":"31ed56c624088b43db9531b58aebe52e19fdaac2e7e9bc3668464af7a82b20c0"} Oct 30 19:47:53.765813 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.765787 6907 scope.go:122] "RemoveContainer" containerID="fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65" Oct 30 19:47:53.766120 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:47:53.766100 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65\": container with ID starting with fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65 not found: ID does not exist" containerID="fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65" Oct 30 19:47:53.766205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.766132 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65"} err="failed to get container status \"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65\": rpc error: code = NotFound desc = could not find container \"fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65\": container with ID starting with fd7db68c71f0025966a2cbaa635940fcd8d55ef6ae11e43344369b02eaa16a65 not found: ID does not exist" Oct 30 19:47:53.766205 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.766156 6907 scope.go:122] "RemoveContainer" containerID="19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec" Oct 30 19:47:53.778204 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.778185 6907 scope.go:122] "RemoveContainer" containerID="19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec" Oct 30 19:47:53.778500 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:47:53.778480 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec\": container with ID starting with 19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec not found: ID does not exist" containerID="19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec" Oct 30 19:47:53.778564 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.778510 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec"} err="failed to get container status \"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec\": rpc error: code = NotFound desc = could not find container \"19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec\": container with ID starting with 19667b81205cb57910ea3d7b459c5c5a2bffd612aaa7417e78abc98cee391cec not found: ID does not exist" Oct 30 19:47:53.783640 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.783603 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:47:53.785590 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.785570 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-7ed3da-6d5d79c6f9-5g96v"] Oct 30 19:47:53.797029 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.797002 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:47:53.801648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.801627 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-7ed3da-745fdfc979-8f5jk"] Oct 30 19:47:53.849231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.849196 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845c010d-c124-4277-b901-ecddb5492814-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:53.849231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.849225 6907 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b04badb-5872-4fdc-aadc-d801fe7db4e1-plugins-config-volume\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:53.849231 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.849236 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzgd5\" (UniqueName: \"kubernetes.io/projected/845c010d-c124-4277-b901-ecddb5492814-kube-api-access-jzgd5\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:53.849434 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:53.849246 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8nfp\" (UniqueName: \"kubernetes.io/projected/6b04badb-5872-4fdc-aadc-d801fe7db4e1-kube-api-access-f8nfp\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:54.150637 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.150601 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" path="/var/lib/kubelet/pods/6b04badb-5872-4fdc-aadc-d801fe7db4e1/volumes" Oct 30 19:47:54.151001 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.150988 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845c010d-c124-4277-b901-ecddb5492814" path="/var/lib/kubelet/pods/845c010d-c124-4277-b901-ecddb5492814/volumes" Oct 30 19:47:54.692058 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.692020 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:47:54.692586 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.692240 6907 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" podUID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" containerName="maas-api" containerID="cri-o://a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4" gracePeriod=30 Oct 30 19:47:54.912746 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.912712 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l"] Oct 30 19:47:54.917383 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.917354 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-7ed3da-bxk5l"] Oct 30 19:47:54.972046 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:54.972005 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:47:55.161765 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.161730 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\") pod \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " Oct 30 19:47:55.161983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.161828 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\") pod \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " Oct 30 19:47:55.161983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.161929 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\" (UniqueName: \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\") pod \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " Oct 30 19:47:55.161983 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.161970 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\") pod \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\" (UID: \"b85be873-35a7-4c6f-afd9-e0246f7efb3a\") " Oct 30 19:47:55.162200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.162176 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca" pod "b85be873-35a7-4c6f-afd9-e0246f7efb3a" (UID: "b85be873-35a7-4c6f-afd9-e0246f7efb3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:55.162365 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.162348 6907 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be873-35a7-4c6f-afd9-e0246f7efb3a-openshift-service-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:55.164026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.164003 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs" pod "b85be873-35a7-4c6f-afd9-e0246f7efb3a" (UID: "b85be873-35a7-4c6f-afd9-e0246f7efb3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:55.164141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.164023 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn" pod "b85be873-35a7-4c6f-afd9-e0246f7efb3a" (UID: "b85be873-35a7-4c6f-afd9-e0246f7efb3a"). InnerVolumeSpecName "kube-api-access-jczqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:55.164141 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.164024 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls" pod "b85be873-35a7-4c6f-afd9-e0246f7efb3a" (UID: "b85be873-35a7-4c6f-afd9-e0246f7efb3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:55.263994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.263893 6907 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-metrics-certs\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:55.263994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.263926 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jczqn\" (UniqueName: \"kubernetes.io/projected/b85be873-35a7-4c6f-afd9-e0246f7efb3a-kube-api-access-jczqn\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:55.263994 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.263941 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be873-35a7-4c6f-afd9-e0246f7efb3a-maas-api-tls\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:55.771720 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.771681 6907 generic.go:356] "Generic (PLEG): container finished" podID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" containerID="a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4" exitCode=0 Oct 30 19:47:55.772208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.771725 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" event={"ID":"b85be873-35a7-4c6f-afd9-e0246f7efb3a","Type":"ContainerDied","Data":"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4"} Oct 30 19:47:55.772208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.771730 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" Oct 30 19:47:55.772208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.771778 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4" event={"ID":"b85be873-35a7-4c6f-afd9-e0246f7efb3a","Type":"ContainerDied","Data":"0395165ed07443d15f1da42392649afb7fc91411f361c1d4cdebb36c45fce8ae"} Oct 30 19:47:55.772208 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.771809 6907 scope.go:122] "RemoveContainer" containerID="a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4" Oct 30 19:47:55.782810 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.782789 6907 scope.go:122] "RemoveContainer" containerID="a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4" Oct 30 19:47:55.783106 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:47:55.783082 6907 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4\": container with ID starting with a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4 not found: ID does not exist" containerID="a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4" Oct 30 19:47:55.783215 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.783111 6907 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4"} err="failed to get container status \"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4\": rpc error: code = NotFound desc = could not find container \"a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4\": container with ID starting with a168865822ccc3876c2f8ba8ce80ce39a5f4fdb90651f0eb343916cb0dae5ff4 not found: ID does not exist" Oct 30 19:47:55.799584 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.799552 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:47:55.803759 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:55.803733 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-7ed3da-8548747754-g88p4"] Oct 30 19:47:56.150736 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:56.150655 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fb04d4-1902-43f6-bb41-36eeef2156d0" path="/var/lib/kubelet/pods/37fb04d4-1902-43f6-bb41-36eeef2156d0/volumes" Oct 30 19:47:56.151078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:56.151065 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" path="/var/lib/kubelet/pods/b85be873-35a7-4c6f-afd9-e0246f7efb3a/volumes" Oct 30 19:47:56.341738 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:47:56.341704 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:48:04.757168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:04.757127 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:48:07.795466 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:48:07.794775 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:48:09.867459 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:09.867423 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:48:14.307688 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.307649 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5"] Oct 30 19:48:14.308221 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308204 6907 cpu_manager.go:414] "containerMap: removing container" podUID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" containerName="payload-processing" Oct 30 19:48:14.308288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308222 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" containerName="payload-processing" Oct 30 19:48:14.308288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308252 6907 cpu_manager.go:414] "containerMap: removing container" podUID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" containerName="maas-api" Oct 30 19:48:14.308288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308257 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" containerName="maas-api" Oct 30 19:48:14.308288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308269 6907 cpu_manager.go:414] "containerMap: removing container" podUID="845c010d-c124-4277-b901-ecddb5492814" containerName="payload-pre-processing" Oct 30 19:48:14.308288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308274 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845c010d-c124-4277-b901-ecddb5492814" containerName="payload-pre-processing" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308293 6907 cpu_manager.go:414] "containerMap: removing container" podUID="37fb04d4-1902-43f6-bb41-36eeef2156d0" containerName="revoke-keys" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308298 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fb04d4-1902-43f6-bb41-36eeef2156d0" containerName="revoke-keys" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308372 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b04badb-5872-4fdc-aadc-d801fe7db4e1" containerName="payload-processing" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308384 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85be873-35a7-4c6f-afd9-e0246f7efb3a" containerName="maas-api" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308393 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="845c010d-c124-4277-b901-ecddb5492814" containerName="payload-pre-processing" Oct 30 19:48:14.308496 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.308400 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fb04d4-1902-43f6-bb41-36eeef2156d0" containerName="revoke-keys" Oct 30 19:48:14.313373 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.313352 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.317972 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.317936 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:14.318114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.317972 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:14.320007 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.319975 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5"] Oct 30 19:48:14.354372 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.354343 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:14.354530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.354406 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kzlqw\"" Oct 30 19:48:14.366474 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.366425 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.366623 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.366555 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.366667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.366636 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzppz\" (UniqueName: \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.467382 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.467341 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzppz\" (UniqueName: \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.467595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.467419 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.467595 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.467519 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.468096 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.468060 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.469830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.469813 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.476359 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.476333 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzppz\" (UniqueName: \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\") pod \"maas-api-revoke-sub-f03629db-qdgk5\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.633257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.633164 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:14.914263 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.914239 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5"] Oct 30 19:48:14.916553 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:48:14.916522 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f83363_e557_44bf_9a1b_a656616ce906.slice/crio-d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4 WatchSource:0}: Error finding container d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4: Status 404 returned error can't find the container with id d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4 Oct 30 19:48:14.935268 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:14.935235 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" event={"ID":"10f83363-e557-44bf-9a1b-a656616ce906","Type":"ContainerStarted","Data":"d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4"} Oct 30 19:48:15.942799 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:15.942762 6907 generic.go:356] "Generic (PLEG): container finished" podID="10f83363-e557-44bf-9a1b-a656616ce906" containerID="2d387c3c6e9b2d4ecb009fd423c54da62bce846c26775ecb7830feda852b08c5" exitCode=0 Oct 30 19:48:15.943197 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:15.942840 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" event={"ID":"10f83363-e557-44bf-9a1b-a656616ce906","Type":"ContainerDied","Data":"2d387c3c6e9b2d4ecb009fd423c54da62bce846c26775ecb7830feda852b08c5"} Oct 30 19:48:17.089320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.089292 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:17.189653 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.189617 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\") pod \"10f83363-e557-44bf-9a1b-a656616ce906\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " Oct 30 19:48:17.189842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.189663 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\" (UniqueName: \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\") pod \"10f83363-e557-44bf-9a1b-a656616ce906\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " Oct 30 19:48:17.189842 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.189755 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\") pod \"10f83363-e557-44bf-9a1b-a656616ce906\" (UID: \"10f83363-e557-44bf-9a1b-a656616ce906\") " Oct 30 19:48:17.190114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.190081 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca" pod "10f83363-e557-44bf-9a1b-a656616ce906" (UID: "10f83363-e557-44bf-9a1b-a656616ce906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:17.191821 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.191800 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca" pod "10f83363-e557-44bf-9a1b-a656616ce906" (UID: "10f83363-e557-44bf-9a1b-a656616ce906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:17.191992 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.191968 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz" pod "10f83363-e557-44bf-9a1b-a656616ce906" (UID: "10f83363-e557-44bf-9a1b-a656616ce906"). InnerVolumeSpecName "kube-api-access-dzppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:17.290786 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.290754 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:17.290953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.290806 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10f83363-e557-44bf-9a1b-a656616ce906-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:17.290953 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.290819 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzppz\" (UniqueName: \"kubernetes.io/projected/10f83363-e557-44bf-9a1b-a656616ce906-kube-api-access-dzppz\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:17.956398 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.956367 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" Oct 30 19:48:17.956607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.956405 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qdgk5" event={"ID":"10f83363-e557-44bf-9a1b-a656616ce906","Type":"ContainerDied","Data":"d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4"} Oct 30 19:48:17.956607 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:17.956465 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ead6902b3dcfd76c3b4726adbfb23cba9d15f402e2560e69f6ba99ad5250f4" Oct 30 19:48:21.033045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:21.033002 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8"] Oct 30 19:48:21.039569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:21.039543 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88dcd18b-rqvp8"] Oct 30 19:48:22.149561 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:22.149523 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0718fdc1-399e-477c-be1f-5af3e2de4b88" path="/var/lib/kubelet/pods/0718fdc1-399e-477c-be1f-5af3e2de4b88/volumes" Oct 30 19:48:25.144619 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:25.144579 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:48:30.446068 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.446031 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm"] Oct 30 19:48:30.446535 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.446520 6907 cpu_manager.go:414] "containerMap: removing container" podUID="10f83363-e557-44bf-9a1b-a656616ce906" containerName="revoke-keys" Oct 30 19:48:30.446588 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.446536 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f83363-e557-44bf-9a1b-a656616ce906" containerName="revoke-keys" Oct 30 19:48:30.446648 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.446638 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f83363-e557-44bf-9a1b-a656616ce906" containerName="revoke-keys" Oct 30 19:48:30.451129 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.451113 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.455248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.455208 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:30.455248 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.455235 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:30.455473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.455235 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:30.455473 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.455245 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kzlqw\"" Oct 30 19:48:30.457593 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.457560 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm"] Oct 30 19:48:30.574811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.574775 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.574811 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.574810 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqlh\" (UniqueName: \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.575030 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.574955 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.675866 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.675821 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.676102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.675920 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.676102 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.675940 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqlh\" (UniqueName: \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.676616 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.676588 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.678237 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.678218 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.690991 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.690958 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqlh\" (UniqueName: \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\") pod \"maas-api-revoke-sub-0165d7d2-qwrgm\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:30.767233 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:30.767203 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:31.043614 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:31.043590 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm"] Oct 30 19:48:31.046124 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:48:31.046086 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d829c0_4a2c_4038_aa73_3b9fc8e2616c.slice/crio-4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9 WatchSource:0}: Error finding container 4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9: Status 404 returned error can't find the container with id 4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9 Oct 30 19:48:32.050539 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:32.050498 6907 generic.go:356] "Generic (PLEG): container finished" podID="40d829c0-4a2c-4038-aa73-3b9fc8e2616c" containerID="97c10334e1827d5ab2e4ef8e84ecac7acef553ec37a1b1779dd90890e9d6450b" exitCode=0 Oct 30 19:48:32.050930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:32.050584 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" event={"ID":"40d829c0-4a2c-4038-aa73-3b9fc8e2616c","Type":"ContainerDied","Data":"97c10334e1827d5ab2e4ef8e84ecac7acef553ec37a1b1779dd90890e9d6450b"} Oct 30 19:48:32.050930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:32.050641 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" event={"ID":"40d829c0-4a2c-4038-aa73-3b9fc8e2616c","Type":"ContainerStarted","Data":"4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9"} Oct 30 19:48:33.202316 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.202290 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:33.300402 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.300364 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\") pod \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " Oct 30 19:48:33.300633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.300426 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\" (UniqueName: \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\") pod \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " Oct 30 19:48:33.300633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.300502 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\") pod \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\" (UID: \"40d829c0-4a2c-4038-aa73-3b9fc8e2616c\") " Oct 30 19:48:33.300843 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.300817 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca" pod "40d829c0-4a2c-4038-aa73-3b9fc8e2616c" (UID: "40d829c0-4a2c-4038-aa73-3b9fc8e2616c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:33.300970 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.300943 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:33.302601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.302576 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca" pod "40d829c0-4a2c-4038-aa73-3b9fc8e2616c" (UID: "40d829c0-4a2c-4038-aa73-3b9fc8e2616c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:33.302680 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.302659 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh" pod "40d829c0-4a2c-4038-aa73-3b9fc8e2616c" (UID: "40d829c0-4a2c-4038-aa73-3b9fc8e2616c"). InnerVolumeSpecName "kube-api-access-qzqlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:33.401844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.401809 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:33.401844 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:33.401839 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqlh\" (UniqueName: \"kubernetes.io/projected/40d829c0-4a2c-4038-aa73-3b9fc8e2616c-kube-api-access-qzqlh\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:34.063581 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:34.063546 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" Oct 30 19:48:34.063581 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:34.063569 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qwrgm" event={"ID":"40d829c0-4a2c-4038-aa73-3b9fc8e2616c","Type":"ContainerDied","Data":"4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9"} Oct 30 19:48:34.063800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:34.063644 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6935dc2ddffc6f3b62f453de962d66efe14f56f2fcd8e30ecab762774e25a9" Oct 30 19:48:37.736480 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:48:37.736372 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:48:41.316365 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.316320 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299"] Oct 30 19:48:41.317063 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.317043 6907 cpu_manager.go:414] "containerMap: removing container" podUID="40d829c0-4a2c-4038-aa73-3b9fc8e2616c" containerName="revoke-keys" Oct 30 19:48:41.317111 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.317064 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d829c0-4a2c-4038-aa73-3b9fc8e2616c" containerName="revoke-keys" Oct 30 19:48:41.317163 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.317144 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d829c0-4a2c-4038-aa73-3b9fc8e2616c" containerName="revoke-keys" Oct 30 19:48:41.320343 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.320327 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.328756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.328726 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299"] Oct 30 19:48:41.365741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.365701 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:41.365741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.365731 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kzlqw\"" Oct 30 19:48:41.374026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.373993 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:48:41.374026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.374009 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:48:41.490490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.490429 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.490490 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.490497 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.490756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.490638 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6k9\" (UniqueName: \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.591873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.591774 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.591873 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.591822 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6k9\" (UniqueName: \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.592134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.592003 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.592624 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.592594 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.594336 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.594311 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.601633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.601613 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6k9\" (UniqueName: \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\") pod \"maas-api-revoke-sub-6b19338f-px299\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.635758 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.635719 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:41.928411 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:41.928390 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299"] Oct 30 19:48:41.931118 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:48:41.931088 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea6f49f_e776_4967_996a_754b9c4f252d.slice/crio-c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce WatchSource:0}: Error finding container c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce: Status 404 returned error can't find the container with id c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce Oct 30 19:48:42.118603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:42.118563 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" event={"ID":"dea6f49f-e776-4967-996a-754b9c4f252d","Type":"ContainerStarted","Data":"c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce"} Oct 30 19:48:43.126556 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:43.126514 6907 generic.go:356] "Generic (PLEG): container finished" podID="dea6f49f-e776-4967-996a-754b9c4f252d" containerID="d2df3ca60b9a2d90c4245357aac8d25aee75db5b3a8f44e9e7dab5bbf3e5d977" exitCode=0 Oct 30 19:48:43.126963 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:43.126647 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" event={"ID":"dea6f49f-e776-4967-996a-754b9c4f252d","Type":"ContainerDied","Data":"d2df3ca60b9a2d90c4245357aac8d25aee75db5b3a8f44e9e7dab5bbf3e5d977"} Oct 30 19:48:44.271516 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.271491 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:44.423290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.423200 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\" (UniqueName: \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\") pod \"dea6f49f-e776-4967-996a-754b9c4f252d\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " Oct 30 19:48:44.423290 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.423259 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\") pod \"dea6f49f-e776-4967-996a-754b9c4f252d\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " Oct 30 19:48:44.423554 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.423331 6907 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\") pod \"dea6f49f-e776-4967-996a-754b9c4f252d\" (UID: \"dea6f49f-e776-4967-996a-754b9c4f252d\") " Oct 30 19:48:44.423677 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.423646 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca" pod "dea6f49f-e776-4967-996a-754b9c4f252d" (UID: "dea6f49f-e776-4967-996a-754b9c4f252d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:44.423825 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.423811 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-openshift-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:44.425345 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.425319 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9" pod "dea6f49f-e776-4967-996a-754b9c4f252d" (UID: "dea6f49f-e776-4967-996a-754b9c4f252d"). InnerVolumeSpecName "kube-api-access-gn6k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:44.425429 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.425328 6907 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca" pod "dea6f49f-e776-4967-996a-754b9c4f252d" (UID: "dea6f49f-e776-4967-996a-754b9c4f252d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:44.524756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.524714 6907 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6k9\" (UniqueName: \"kubernetes.io/projected/dea6f49f-e776-4967-996a-754b9c4f252d-kube-api-access-gn6k9\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:44.524756 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:44.524751 6907 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea6f49f-e776-4967-996a-754b9c4f252d-maas-api-xks-ca\") on node \"ip-10-0-132-153.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:45.139004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:45.138971 6907 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" Oct 30 19:48:45.139004 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:45.138988 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-px299" event={"ID":"dea6f49f-e776-4967-996a-754b9c4f252d","Type":"ContainerDied","Data":"c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce"} Oct 30 19:48:45.139222 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:45.139032 6907 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17aedab7fcd6de4debef741eec4fa3616efbada021eec6072a900dee1ccc6ce" Oct 30 19:48:52.284608 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:48:52.284571 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-749qf"] Oct 30 19:49:03.434317 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:03.434284 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-689bf9d68c-8qzns_303ef8a0-c0c7-4749-a661-6f95f42348c3/manager/0.log" Oct 30 19:49:04.274168 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:04.274134 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:49:04.494631 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:04.494598 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wfznm_77cf09e2-a980-402f-9685-9727db0b70ee/server/0.log" Oct 30 19:49:04.676890 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:04.676795 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hnbrx_ec4c3328-a427-4baa-9205-6cd30fcc5770/manager/0.log" Oct 30 19:49:06.032667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.032631 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qwrgm_40d829c0-4a2c-4038-aa73-3b9fc8e2616c/revoke-keys/0.log" Oct 30 19:49:06.205014 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.204988 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-233ab402-5xgc7_b0cc575e-0762-409e-85ec-b70a44467b0a/revoke-keys/0.log" Oct 30 19:49:06.383188 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.383113 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-ps87n_38e8da9b-783a-4a7a-a507-935e49e66b4f/revoke-keys/0.log" Oct 30 19:49:06.578820 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.578789 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gf2xl_dad4c916-4abf-46e1-b600-62c8d622dfde/revoke-keys/0.log" Oct 30 19:49:06.762169 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.762143 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-px299_dea6f49f-e776-4967-996a-754b9c4f252d/revoke-keys/0.log" Oct 30 19:49:06.945689 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:06.945659 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qdgk5_10f83363-e557-44bf-9a1b-a656616ce906/revoke-keys/0.log" Oct 30 19:49:07.737335 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:49:07.737295 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:49:09.805053 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:09.805021 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-749qf_afc2a5cc-51af-4df2-ba60-5dae4e376238/limitador/0.log" Oct 30 19:49:09.997370 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:09.997339 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b8282_f691bc00-6099-461b-a432-61ee33696e55/manager/0.log" Oct 30 19:49:10.955417 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:10.955386 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ld9n2_75c0004a-8bf5-4cb7-9d05-411d92b85476/discovery/0.log" Oct 30 19:49:11.328803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:11.328753 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6944dd5f6b-cz7g6_88351064-cc56-4128-a7af-708e83790454/kube-auth-proxy/0.log" Oct 30 19:49:11.876734 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:11.876702 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/1.log" Oct 30 19:49:12.403694 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:12.403660 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pns2g_3a9a5b0e-97f2-495a-89e7-516b96f2887f/istio-proxy/0.log" Oct 30 19:49:12.580238 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:12.580208 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rp9rd_147200da-9ad7-4a69-b764-e074edb12c9f/istio-proxy/0.log" Oct 30 19:49:13.199103 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:13.199072 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-vzsw7_0818d712-9d8b-4db5-9b05-ffca59cfdea2/istio-proxy/0.log" Oct 30 19:49:13.591269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:13.591237 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ddzlh_0a76f54a-2ca7-401b-86a0-7e51f33fced3/payload-pre-processing/0.log" Oct 30 19:49:13.779200 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:13.779169 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:49:14.482501 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:14.482469 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/0.log" Oct 30 19:49:15.049944 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:15.049912 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w_c52a7fed-385d-45b9-a1a4-78be7a912e60/main/0.log" Oct 30 19:49:26.715269 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:26.715233 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-689bf9d68c-8qzns_303ef8a0-c0c7-4749-a661-6f95f42348c3/manager/0.log" Oct 30 19:49:27.554422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:27.554388 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:49:27.761930 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:27.761902 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wfznm_77cf09e2-a980-402f-9685-9727db0b70ee/server/0.log" Oct 30 19:49:27.949601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:27.949516 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-hnbrx_ec4c3328-a427-4baa-9205-6cd30fcc5770/manager/0.log" Oct 30 19:49:29.093908 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.093870 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qwrgm_40d829c0-4a2c-4038-aa73-3b9fc8e2616c/revoke-keys/0.log" Oct 30 19:49:29.273927 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.273899 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-233ab402-5xgc7_b0cc575e-0762-409e-85ec-b70a44467b0a/revoke-keys/0.log" Oct 30 19:49:29.442605 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.442523 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-ps87n_38e8da9b-783a-4a7a-a507-935e49e66b4f/revoke-keys/0.log" Oct 30 19:49:29.622114 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.622081 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-gf2xl_dad4c916-4abf-46e1-b600-62c8d622dfde/revoke-keys/0.log" Oct 30 19:49:29.807363 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.807329 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-px299_dea6f49f-e776-4967-996a-754b9c4f252d/revoke-keys/0.log" Oct 30 19:49:29.992282 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:29.992250 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-qdgk5_10f83363-e557-44bf-9a1b-a656616ce906/revoke-keys/0.log" Oct 30 19:49:31.024277 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:31.024244 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl"] Oct 30 19:49:31.027184 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:31.027159 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-gf2xl"] Oct 30 19:49:32.150209 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:32.150177 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad4c916-4abf-46e1-b600-62c8d622dfde" path="/var/lib/kubelet/pods/dad4c916-4abf-46e1-b600-62c8d622dfde/volumes" Oct 30 19:49:32.379793 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:32.379762 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-749qf_afc2a5cc-51af-4df2-ba60-5dae4e376238/limitador/0.log" Oct 30 19:49:32.568098 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:32.568069 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b8282_f691bc00-6099-461b-a432-61ee33696e55/manager/0.log" Oct 30 19:49:33.298889 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:33.298854 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ld9n2_75c0004a-8bf5-4cb7-9d05-411d92b85476/discovery/0.log" Oct 30 19:49:33.678780 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:33.678691 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6944dd5f6b-cz7g6_88351064-cc56-4128-a7af-708e83790454/kube-auth-proxy/0.log" Oct 30 19:49:34.222667 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:34.222634 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/1.log" Oct 30 19:49:34.770479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:34.770434 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pns2g_3a9a5b0e-97f2-495a-89e7-516b96f2887f/istio-proxy/0.log" Oct 30 19:49:34.953479 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:34.953433 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rp9rd_147200da-9ad7-4a69-b764-e074edb12c9f/istio-proxy/0.log" Oct 30 19:49:35.477976 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:35.477943 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-vzsw7_0818d712-9d8b-4db5-9b05-ffca59cfdea2/istio-proxy/0.log" Oct 30 19:49:35.837580 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:35.837492 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ddzlh_0a76f54a-2ca7-401b-86a0-7e51f33fced3/payload-pre-processing/0.log" Oct 30 19:49:36.037748 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:36.037719 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:49:36.754665 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:36.754629 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-rlt2w_c52a7fed-385d-45b9-a1a4-78be7a912e60/main/0.log" Oct 30 19:49:37.736201 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:49:37.736155 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:49:47.040153 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:47.040111 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n"] Oct 30 19:49:47.043784 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:47.043756 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-ps87n"] Oct 30 19:49:48.150278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:48.150241 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e8da9b-783a-4a7a-a507-935e49e66b4f" path="/var/lib/kubelet/pods/38e8da9b-783a-4a7a-a507-935e49e66b4f/volumes" Oct 30 19:49:54.853313 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:54.853286 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:49:54.872059 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:54.872034 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:49:54.911207 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:54.911178 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ccrzh_5669578a-f791-4363-b718-5339a4399bd9/extract/0.log" Oct 30 19:49:54.931241 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:54.931213 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wv29b_f92253fd-8219-46e9-b7c6-a4ccebbd7dd4/ovn-acl-logging/0.log" Oct 30 19:49:57.552130 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:57.552096 6907 scope.go:122] "RemoveContainer" containerID="7bb155e51971f7137a52199a1e4323cae5eadc7e39dc30ab36cd4a864b503a26" Oct 30 19:49:57.561753 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:49:57.561733 6907 scope.go:122] "RemoveContainer" containerID="bf20df480c2baa2894fb4331b40dfe4505d5a4f625cb6826520b9e21df545311" Oct 30 19:50:06.025461 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:06.025343 6907 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7"] Oct 30 19:50:06.027379 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:06.027355 6907 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-233ab402-5xgc7"] Oct 30 19:50:06.150173 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:06.150138 6907 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cc575e-0762-409e-85ec-b70a44467b0a" path="/var/lib/kubelet/pods/b0cc575e-0762-409e-85ec-b70a44467b0a/volumes" Oct 30 19:50:07.736099 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:50:07.736057 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:50:16.845149 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:16.845113 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:50:17.143899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:17.143819 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:50:17.872311 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:17.872248 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:50:18.053171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:18.053140 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:50:37.736089 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:50:37.736052 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:50:57.589465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:57.589409 6907 scope.go:122] "RemoveContainer" containerID="c35970b648b58482a1a2aaac1d7a6d2a801a18754c92a9de6beb39cd136861dc" Oct 30 19:50:57.599862 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:50:57.599802 6907 scope.go:122] "RemoveContainer" containerID="74ad7d63797378517cda691f63030199442c4e5e7d60c2774aab5aca3e65915e" Oct 30 19:51:04.832203 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:04.832168 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:51:05.094109 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:05.094023 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78455b774c-7466n_d618490d-5e22-4274-833c-ab8d9b95453e/manager/0.log" Oct 30 19:51:05.839741 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:05.839714 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:51:06.019307 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:06.019276 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:51:07.736396 ip-10-0-132-153 kubenswrapper[6907]: E1030 19:51:07.736354 6907 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fv898" containerName="extractor" Oct 30 19:51:09.607220 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:09.607192 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n84k8_4738657b-e0c8-4a35-9893-85f99f3632b2/global-pull-secret-syncer/0.log" Oct 30 19:51:09.791830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:09.791796 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h9jxz_3e8b6a1e-6af3-4e39-9f3b-7b05da2fbf23/konnectivity-agent/0.log" Oct 30 19:51:09.936125 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:09.936041 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-153.us-east-2.compute.internal_1ae4921b5faf0892072f47ac728ceb26/haproxy/0.log" Oct 30 19:51:17.333026 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:17.332988 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-749qf_afc2a5cc-51af-4df2-ba60-5dae4e376238/limitador/0.log" Oct 30 19:51:17.414596 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:17.414553 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-b8282_f691bc00-6099-461b-a432-61ee33696e55/manager/0.log" Oct 30 19:51:18.614458 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.614414 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/alertmanager/0.log" Oct 30 19:51:18.641940 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.641911 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/config-reloader/0.log" Oct 30 19:51:18.678045 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.678017 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/kube-rbac-proxy-web/0.log" Oct 30 19:51:18.704768 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.704742 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/kube-rbac-proxy/0.log" Oct 30 19:51:18.735729 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.735704 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/kube-rbac-proxy-metric/0.log" Oct 30 19:51:18.765086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.765057 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/prom-label-proxy/0.log" Oct 30 19:51:18.792583 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.792555 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20049954-3de4-452c-a17d-351680769c2e/init-config-reloader/0.log" Oct 30 19:51:18.865917 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.865845 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xwhmz_2aaa7bfb-b228-4955-9670-bb51ae187fed/kube-state-metrics/0.log" Oct 30 19:51:18.887929 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.887899 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xwhmz_2aaa7bfb-b228-4955-9670-bb51ae187fed/kube-rbac-proxy-main/0.log" Oct 30 19:51:18.914727 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.914700 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-xwhmz_2aaa7bfb-b228-4955-9670-bb51ae187fed/kube-rbac-proxy-self/0.log" Oct 30 19:51:18.944134 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.944106 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6c5b8d6bd7-rp9k6_33d3aaac-34ae-41ca-883b-f4b6e0bdd642/metrics-server/0.log" Oct 30 19:51:18.980717 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:18.980689 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-snf6t_92b270d8-4b48-48c9-a930-99d9c35419cc/monitoring-plugin/0.log" Oct 30 19:51:19.031172 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.031139 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5rpc_2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/node-exporter/0.log" Oct 30 19:51:19.059387 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.059357 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5rpc_2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/kube-rbac-proxy/0.log" Oct 30 19:51:19.084959 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.084936 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5rpc_2ce8d5b4-6c1d-46e1-8e02-49cb400c05f0/init-textfile/0.log" Oct 30 19:51:19.322426 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.322398 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8g6jq_f10dec5f-455a-4587-8c73-6f9c872a0372/kube-rbac-proxy-main/0.log" Oct 30 19:51:19.350987 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.350957 6907 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk"] Oct 30 19:51:19.351439 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.351425 6907 cpu_manager.go:414] "containerMap: removing container" podUID="dea6f49f-e776-4967-996a-754b9c4f252d" containerName="revoke-keys" Oct 30 19:51:19.351504 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.351452 6907 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea6f49f-e776-4967-996a-754b9c4f252d" containerName="revoke-keys" Oct 30 19:51:19.351579 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.351569 6907 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea6f49f-e776-4967-996a-754b9c4f252d" containerName="revoke-keys" Oct 30 19:51:19.355570 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.355550 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.365086 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.365058 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8mgz\"/\"kube-root-ca.crt\"" Oct 30 19:51:19.365244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.365167 6907 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d8mgz\"/\"default-dockercfg-8p9jx\"" Oct 30 19:51:19.365244 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.365233 6907 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8mgz\"/\"openshift-service-ca.crt\"" Oct 30 19:51:19.366681 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.366660 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8g6jq_f10dec5f-455a-4587-8c73-6f9c872a0372/kube-rbac-proxy-self/0.log" Oct 30 19:51:19.367830 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.367807 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk"] Oct 30 19:51:19.402673 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.402646 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8g6jq_f10dec5f-455a-4587-8c73-6f9c872a0372/openshift-state-metrics/0.log" Oct 30 19:51:19.452848 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.452815 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/prometheus/0.log" Oct 30 19:51:19.475332 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.475301 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-lib-modules\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.475520 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.475348 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7675w\" (UniqueName: \"kubernetes.io/projected/f997d659-074f-4703-84f6-f32136dc26be-kube-api-access-7675w\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.475574 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.475511 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-podres\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.475574 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.475545 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-sys\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.475658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.475638 6907 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-proc\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.482958 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.482927 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/config-reloader/0.log" Oct 30 19:51:19.512800 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.512779 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/thanos-sidecar/0.log" Oct 30 19:51:19.544073 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.544048 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/kube-rbac-proxy-web/0.log" Oct 30 19:51:19.574658 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.574585 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/kube-rbac-proxy/0.log" Oct 30 19:51:19.576961 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.576939 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-podres\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577022 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.576973 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-sys\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577020 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-proc\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577066 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577053 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-lib-modules\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577058 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-podres\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577104 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-proc\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577128 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-lib-modules\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577171 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577129 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f997d659-074f-4703-84f6-f32136dc26be-sys\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.577386 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.577170 6907 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7675w\" (UniqueName: \"kubernetes.io/projected/f997d659-074f-4703-84f6-f32136dc26be-kube-api-access-7675w\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.588700 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.588673 6907 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7675w\" (UniqueName: \"kubernetes.io/projected/f997d659-074f-4703-84f6-f32136dc26be-kube-api-access-7675w\") pod \"perf-node-gather-daemonset-vcnvk\" (UID: \"f997d659-074f-4703-84f6-f32136dc26be\") " pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.608803 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.608775 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/kube-rbac-proxy-thanos/0.log" Oct 30 19:51:19.635721 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.635698 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_689f237d-9e75-4ba8-a205-c7bc295bf7a8/init-config-reloader/0.log" Oct 30 19:51:19.671051 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.671017 6907 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:19.906973 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.906884 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/thanos-query/0.log" Oct 30 19:51:19.938534 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.938502 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/kube-rbac-proxy-web/0.log" Oct 30 19:51:19.949886 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.949853 6907 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk"] Oct 30 19:51:19.952328 ip-10-0-132-153 kubenswrapper[6907]: W1030 19:51:19.952303 6907 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf997d659_074f_4703_84f6_f32136dc26be.slice/crio-95e194ff9777090508750d77751ae021844d435bf313004d388f55cb464c38e6 WatchSource:0}: Error finding container 95e194ff9777090508750d77751ae021844d435bf313004d388f55cb464c38e6: Status 404 returned error can't find the container with id 95e194ff9777090508750d77751ae021844d435bf313004d388f55cb464c38e6 Oct 30 19:51:19.979043 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:19.979023 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/kube-rbac-proxy/0.log" Oct 30 19:51:20.010537 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.010518 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/prom-label-proxy/0.log" Oct 30 19:51:20.035907 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.035876 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/kube-rbac-proxy-rules/0.log" Oct 30 19:51:20.055933 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.055904 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" event={"ID":"f997d659-074f-4703-84f6-f32136dc26be","Type":"ContainerStarted","Data":"a26b453f17899539b81daf0100c9460107c9934395bbc7d20a6db49ddeb6644a"} Oct 30 19:51:20.056078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.055963 6907 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" event={"ID":"f997d659-074f-4703-84f6-f32136dc26be","Type":"ContainerStarted","Data":"95e194ff9777090508750d77751ae021844d435bf313004d388f55cb464c38e6"} Oct 30 19:51:20.056078 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.055999 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:20.064172 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.064151 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dff9b6c5c-ghxcg_306fb251-67bf-402f-a8a4-2da67c1b2013/kube-rbac-proxy-metrics/0.log" Oct 30 19:51:20.075000 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:20.074960 6907 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" podStartSLOduration=1.074947533 podStartE2EDuration="1.074947533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:20.072972373 +0000 UTC m=+1886.049198318" watchObservedRunningTime="2026-10-30 19:51:20.074947533 +0000 UTC m=+1886.051173470" Oct 30 19:51:24.573569 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:24.573537 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxxbw_5364bcd9-a94f-4072-973c-bc3d9993d2bd/dns/0.log" Oct 30 19:51:24.599531 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:24.599486 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dxxbw_5364bcd9-a94f-4072-973c-bc3d9993d2bd/kube-rbac-proxy/0.log" Oct 30 19:51:24.688632 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:24.688604 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kvp2_cb207c2c-2bf8-4263-b349-aa2f5be7b0fb/dns-node-resolver/0.log" Oct 30 19:51:25.440988 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:25.440958 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxnr9_a6dd1ff4-96b4-430d-b31c-ef3a77fe9f18/node-ca/0.log" Oct 30 19:51:26.072278 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:26.072251 6907 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d8mgz/perf-node-gather-daemonset-vcnvk" Oct 30 19:51:28.044563 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.044528 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ld9n2_75c0004a-8bf5-4cb7-9d05-411d92b85476/discovery/0.log" Oct 30 19:51:28.149092 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.149062 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6944dd5f6b-cz7g6_88351064-cc56-4128-a7af-708e83790454/kube-auth-proxy/0.log" Oct 30 19:51:28.268744 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.268682 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/0.log" Oct 30 19:51:28.269422 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.269396 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/1.log" Oct 30 19:51:28.376257 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.376177 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pns2g_3a9a5b0e-97f2-495a-89e7-516b96f2887f/istio-proxy/0.log" Oct 30 19:51:28.413423 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.413395 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rp9rd_147200da-9ad7-4a69-b764-e074edb12c9f/istio-proxy/0.log" Oct 30 19:51:28.530603 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.530574 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-vzsw7_0818d712-9d8b-4db5-9b05-ffca59cfdea2/istio-proxy/0.log" Oct 30 19:51:28.613965 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.613929 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ddzlh_0a76f54a-2ca7-401b-86a0-7e51f33fced3/payload-pre-processing/0.log" Oct 30 19:51:28.645297 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:28.645193 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:51:30.431794 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:30.431759 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrv7f_39f57a2e-06f1-4918-a6b3-e93802969d3d/serve-healthcheck-canary/0.log" Oct 30 19:51:31.276137 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.276105 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ld9n2_75c0004a-8bf5-4cb7-9d05-411d92b85476/discovery/0.log" Oct 30 19:51:31.313530 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.313502 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6944dd5f6b-cz7g6_88351064-cc56-4128-a7af-708e83790454/kube-auth-proxy/0.log" Oct 30 19:51:31.396296 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.396247 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/0.log" Oct 30 19:51:31.396899 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.396882 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/1.log" Oct 30 19:51:31.471713 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.471675 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fv898_fa53f8d4-8928-4045-b84f-9d92c83c66e2/kube-rbac-proxy/0.log" Oct 30 19:51:31.474145 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.474124 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pns2g_3a9a5b0e-97f2-495a-89e7-516b96f2887f/istio-proxy/0.log" Oct 30 19:51:31.508989 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.508961 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rp9rd_147200da-9ad7-4a69-b764-e074edb12c9f/istio-proxy/0.log" Oct 30 19:51:31.516979 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.516956 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fv898_fa53f8d4-8928-4045-b84f-9d92c83c66e2/exporter/0.log" Oct 30 19:51:31.572521 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.572417 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fv898_fa53f8d4-8928-4045-b84f-9d92c83c66e2/extractor/0.log" Oct 30 19:51:31.611288 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.611250 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-vzsw7_0818d712-9d8b-4db5-9b05-ffca59cfdea2/istio-proxy/0.log" Oct 30 19:51:31.663601 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.663572 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ddzlh_0a76f54a-2ca7-401b-86a0-7e51f33fced3/payload-pre-processing/0.log" Oct 30 19:51:31.707192 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:31.707152 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log" Oct 30 19:51:34.615465 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:34.615416 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-668f68f55d-mzw7b_0032836d-a143-44b5-a034-249e94c41a76/manager/0.log" Oct 30 19:51:34.647934 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:34.647902 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-668f68f55d-mzw7b_0032836d-a143-44b5-a034-249e94c41a76/copy-manifests/0.log" Oct 30 19:51:34.944179 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:34.944079 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ld9n2_75c0004a-8bf5-4cb7-9d05-411d92b85476/discovery/0.log" Oct 30 19:51:34.999320 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:34.999288 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6944dd5f6b-cz7g6_88351064-cc56-4128-a7af-708e83790454/kube-auth-proxy/0.log" Oct 30 19:51:35.071685 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.071635 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/0.log" Oct 30 19:51:35.072750 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.072723 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-bcxng_d29f3ff5-991d-4ca7-af41-a592d6e767da/istio-proxy/1.log" Oct 30 19:51:35.131633 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.131587 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-pns2g_3a9a5b0e-97f2-495a-89e7-516b96f2887f/istio-proxy/0.log" Oct 30 19:51:35.150250 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.150219 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-rp9rd_147200da-9ad7-4a69-b764-e074edb12c9f/istio-proxy/0.log" Oct 30 19:51:35.210771 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.210686 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75bf867b6c-vzsw7_0818d712-9d8b-4db5-9b05-ffca59cfdea2/istio-proxy/0.log" Oct 30 19:51:35.257403 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.257376 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-ddzlh_0a76f54a-2ca7-401b-86a0-7e51f33fced3/payload-pre-processing/0.log" Oct 30 19:51:35.274289 ip-10-0-132-153 kubenswrapper[6907]: I1030 19:51:35.274261 6907 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-pp54j_a01ba900-adf8-476f-9d76-08ac8ec45ca0/payload-processing/0.log"